[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:29 dist-test-slave-wvpw krb5kdc[20839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:29 dist-test-slave-wvpw krb5kdc[20839](info): set up 2 sockets
Sep 28 23:54:29 dist-test-slave-wvpw krb5kdc[20839](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 23:54:30.168949 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:30 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:30.299914 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36961
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:36961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:30.421537 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:30.421941 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:30.422016 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:30.426553 20852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:30.426666 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:30.426702 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:30.426744 20852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:30.426781 20852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:30.432732 20852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:36961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:30.434011 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:30.435184 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:30.446090 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:30.446080 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:30.446048 20852 server_base.cc:1034] running on GCE node
W20260928 23:54:30.446125 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:30.446981 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:30.448474 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:30.449684 20852 hybrid_clock.cc:648] HybridClock initialized: now 1790639670449668 us; error 34 us; skew 500 ppm
Sep 28 23:54:30 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639670, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:30.467218 20852 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:30.468561 20852 webserver.cc:460] Webserver started at http://127.20.88.62:34751/ using document root <none> and password file <none>
I20260928 23:54:30.469390 20852 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:30.469480 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:30.469764 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:30.472393 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "d217d58c43834574b2bc10bf11aafb8b"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "83391c44418d1142dfc5f1d5f182b40a"
server_key_iv: "46c8783f30e6119c1b5f717f5edd8cdc"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.473119 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "d217d58c43834574b2bc10bf11aafb8b"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "83391c44418d1142dfc5f1d5f182b40a"
server_key_iv: "46c8783f30e6119c1b5f717f5edd8cdc"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.477684 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 23:54:30.480927 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:30.482575 20852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:54:30.482851 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "d217d58c43834574b2bc10bf11aafb8b"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "83391c44418d1142dfc5f1d5f182b40a"
server_key_iv: "46c8783f30e6119c1b5f717f5edd8cdc"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.483007 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:30.496119 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:30.500065 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:30.500387 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:30.508998 20852 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:36961
I20260928 23:54:30.509013 20919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:36961 every 8 connection(s)
I20260928 23:54:30.510079 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:30.513151 20920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:30.518473 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 20852
I20260928 23:54:30.518661 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:30.520357 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b: Bootstrap starting.
I20260928 23:54:30.520478 20832 external_mini_cluster.cc:1428] Setting key a913366e6ba73b68f5efdbffdba89e20
I20260928 23:54:30.523744 20920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:30.524719 20920 log.cc:824] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:30.527402 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b: No bootstrap required, opened a new log
I20260928 23:54:30.532430 20920 raft_consensus.cc:348] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } }
I20260928 23:54:30.532764 20920 raft_consensus.cc:374] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:30.532909 20920 raft_consensus.cc:729] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d217d58c43834574b2bc10bf11aafb8b, State: Initialized, Role: FOLLOWER
I20260928 23:54:30.533478 20920 consensus_queue.cc:260] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } }
I20260928 23:54:30.533666 20920 raft_consensus.cc:388] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:30.533735 20920 raft_consensus.cc:482] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:30.533862 20920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:30.535290 20920 raft_consensus.cc:504] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } }
I20260928 23:54:30.535708 20920 leader_election.cc:302] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d217d58c43834574b2bc10bf11aafb8b; no voters:
I20260928 23:54:30.536134 20920 leader_election.cc:288] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:30.536298 20925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:30.536535 20925 raft_consensus.cc:686] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [term 1 LEADER]: Becoming Leader. State: Replica: d217d58c43834574b2bc10bf11aafb8b, State: Running, Role: LEADER
I20260928 23:54:30.536870 20925 consensus_queue.cc:237] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } }
I20260928 23:54:30.537629 20920 sys_catalog.cc:564] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:30.538679 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d217d58c43834574b2bc10bf11aafb8b. Latest consensus state: current_term: 1 leader_uuid: "d217d58c43834574b2bc10bf11aafb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } } }
I20260928 23:54:30.538830 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:30.539227 20934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:30.540017 20926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d217d58c43834574b2bc10bf11aafb8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d217d58c43834574b2bc10bf11aafb8b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36961 } } }
I20260928 23:54:30.540113 20926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:30.543335 20934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:54:30 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:30.550022 20934 catalog_manager.cc:1347] Generated new cluster ID: bfc58ee68027401f97c3a9ba6910f646
I20260928 23:54:30.550171 20934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:30.553452 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:30.524279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.20.88.62:36961)
0928 23:54:30.524865 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:30.524887 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:30.526264 (+ 1377us) server_negotiation.cc:407] Connection header received
0928 23:54:30.531877 (+ 5613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:30.531899 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:30.532406 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:30.532770 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:30.536072 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.537690 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.539720 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.540657 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.540852 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:30.546407 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:30.546415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:30.546437 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:30.546470 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:30.548910 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:30.549684 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:30.549693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:30.549702 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:30.549856 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:30.550322 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:30.550326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:30.550328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:30.550888 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:30.551094 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:30.551977 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 23:54:30.552357 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":179,"threads_started":1}
I20260928 23:54:30.559815 20934 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:30.561316 20934 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:30.572434 20934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d217d58c43834574b2bc10bf11aafb8b: Generated new TSK 0
I20260928 23:54:30.573542 20934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:30.612476 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:30.733558 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:30.733951 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:30.734156 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:30.738982 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:30.739094 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:30.739195 20946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:30.745175 20946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:30.746450 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:30.747731 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:30.758632 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:30.758641 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:30.758636 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:30.759123 20946 server_base.cc:1034] running on GCE node
I20260928 23:54:30.759666 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:30.760468 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:30.761696 20946 hybrid_clock.cc:648] HybridClock initialized: now 1790639670761651 us; error 54 us; skew 500 ppm
Sep 28 23:54:30 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639670, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:30.764911 20946 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:30.765914 20946 webserver.cc:460] Webserver started at http://127.20.88.1:35579/ using document root <none> and password file <none>
I20260928 23:54:30.766561 20946 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:30.766662 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:30.766918 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:30.769290 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "705d2fa3dda442deb64f3711de6cfd07"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "569fadb29ff3ed788b70e5fe3ae3122d"
server_key_iv: "1a0732b07aefee396b478b77888ea301"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.769915 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "705d2fa3dda442deb64f3711de6cfd07"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "569fadb29ff3ed788b70e5fe3ae3122d"
server_key_iv: "1a0732b07aefee396b478b77888ea301"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.774255 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:30.777132 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:30.778605 20946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:30.778798 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "705d2fa3dda442deb64f3711de6cfd07"
format_stamp: "Formatted at 2026-09-28 23:54:30 on dist-test-slave-wvpw"
server_key: "569fadb29ff3ed788b70e5fe3ae3122d"
server_key_iv: "1a0732b07aefee396b478b77888ea301"
server_key_version: "encryptionkey@0"
I20260928 23:54:30.778950 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:30.812762 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:30.816368 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:30.816605 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:30.817317 20946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:30.818298 20946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:30.818383 20946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:30.818470 20946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:30.818517 20946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:30.829170 20946 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39811
I20260928 23:54:30.829193 21074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39811 every 8 connection(s)
I20260928 23:54:30.830256 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:30.831174 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 20946
I20260928 23:54:30.831290 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:30.831555 20832 external_mini_cluster.cc:1428] Setting key 7cb58798b5d9c752a15acfd410c93807
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Sep 28 23:54:30 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639670, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:30.849017 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:30.832628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:46503 (local address 127.20.88.62:36961)
0928 23:54:30.832784 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:30.832789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:30.833962 (+ 1173us) server_negotiation.cc:407] Connection header received
0928 23:54:30.836012 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:30.836018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:30.836092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:30.836203 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:30.838270 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.839298 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.840853 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.841262 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.841417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:30.844443 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:30.844447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:30.844449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:30.844480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:30.846000 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:30.846815 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:30.846822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:30.846827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:30.846939 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:30.847429 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:30.847432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:30.847433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:30.847666 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:30.847755 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:30.848607 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 23:54:30.848787 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:54:30.849982 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:30.833092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36961 (local address 127.20.88.1:46503)
0928 23:54:30.833760 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:30.833833 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:30.835731 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:30.836524 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:30.836539 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:30.837230 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:30.837923 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:30.837944 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.839509 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.839512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:30.840693 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:30.840707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:30.841499 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:30.841504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:30.841732 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:30.842457 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:30.842478 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:30.844140 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:30.846139 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:30.846146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:30.846163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:30.846600 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:30.847066 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:30.847072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:30.847077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:30.847311 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:30.847787 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:30.847797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:30.848210 (+ 413us) client_negotiation.cc:769] Sending connection context
0928 23:54:30.848488 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 23:54:30.848882 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":175,"threads_started":1}
I20260928 23:54:30.851426 21075 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36961
I20260928 23:54:30.851722 21075 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:30.852377 21075 heartbeater.cc:511] Master 127.20.88.62:36961 requested a full tablet report, sending...
I20260928 23:54:30.854333 20884 ts_manager.cc:194] Registered new tserver with Master: 705d2fa3dda442deb64f3711de6cfd07 (127.20.88.1:39811)
I20260928 23:54:30.855589 20884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:46503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:30.869480 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:30.992522 21080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:30.992838 21080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:30.992887 21080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:30.997985 21080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:30.998097 21080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:30.998168 21080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:31.005005 21080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:31.006306 21080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:31.007491 21080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:54:31.019716 21080 server_base.cc:1034] running on GCE node
W20260928 23:54:31.019981 21088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:31.019694 21086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:31.019694 21085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:31.020774 21080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:31.021553 21080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:31.022781 21080 hybrid_clock.cc:648] HybridClock initialized: now 1790639671022764 us; error 37 us; skew 500 ppm
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:31.026259 21080 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:31.027340 21080 webserver.cc:460] Webserver started at http://127.20.88.2:41637/ using document root <none> and password file <none>
I20260928 23:54:31.027997 21080 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:31.028107 21080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:31.028465 21080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:31.031201 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "99a44ed0c91d486a9bc740d128de683c"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "ff6914089fd43b47aeb027fef35660b4"
server_key_iv: "92b32e1ee06fcb73f99672fcb1da7f12"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.031903 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "99a44ed0c91d486a9bc740d128de683c"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "ff6914089fd43b47aeb027fef35660b4"
server_key_iv: "92b32e1ee06fcb73f99672fcb1da7f12"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.036435 21080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:54:31.039408 21095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.041018 21080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:31.041167 21080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "99a44ed0c91d486a9bc740d128de683c"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "ff6914089fd43b47aeb027fef35660b4"
server_key_iv: "92b32e1ee06fcb73f99672fcb1da7f12"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.041297 21080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:31.061285 21080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:31.064798 21080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:31.065034 21080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:31.065693 21080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:31.066613 21080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:31.066663 21080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.066730 21080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:31.066766 21080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.077224 21080 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:37895
I20260928 23:54:31.077255 21208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:37895 every 8 connection(s)
I20260928 23:54:31.078270 21080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:31.082388 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21080
I20260928 23:54:31.082526 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:31.082818 20832 external_mini_cluster.cc:1428] Setting key d5433e22b5fe116d849a0dd4d97c4a9e
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:54:31.094807 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.080353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38651 (local address 127.20.88.62:36961)
0928 23:54:31.080526 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:31.080530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:31.081517 (+ 987us) server_negotiation.cc:407] Connection header received
0928 23:54:31.083271 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:31.083275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:31.083361 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:31.083500 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:31.084972 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.085536 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.086887 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.087150 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.087272 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.090220 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:31.090224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:31.090226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:31.090257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:31.092039 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.092713 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.092725 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.092727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.092817 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.093347 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.093352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.093354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.093580 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:31.093699 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:31.094376 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 23:54:31.094544 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:54:31.095770 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.080803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36961 (local address 127.20.88.2:38651)
0928 23:54:31.081324 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:31.081383 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:31.083019 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:31.083722 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:31.083730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:31.084154 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:31.084800 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:31.084815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.085682 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.085693 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:31.086703 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:31.086714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.087335 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.087340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:31.087538 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.088333 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:31.088354 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:31.089996 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:31.092196 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:31.092203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:31.092225 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:31.092579 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:31.092939 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:31.092944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:31.092946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:31.093217 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:31.093709 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:31.093720 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:31.094071 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 23:54:31.094328 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 23:54:31.094730 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260928 23:54:31.097543 21209 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36961
I20260928 23:54:31.097926 21209 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:31.098517 21209 heartbeater.cc:511] Master 127.20.88.62:36961 requested a full tablet report, sending...
I20260928 23:54:31.099812 20884 ts_manager.cc:194] Registered new tserver with Master: 99a44ed0c91d486a9bc740d128de683c (127.20.88.2:37895)
I20260928 23:54:31.100543 20884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:38651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:31.111676 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:31.230253 21214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:31.230600 21214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:31.230648 21214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:31.234891 21214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:31.234978 21214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:31.235044 21214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:31.241153 21214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36961
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:31.242465 21214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:31.243566 21214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:31.254490 21219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:31.254544 21214 server_base.cc:1034] running on GCE node
W20260928 23:54:31.254465 21220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:31.254465 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:31.255270 21214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:31.255965 21214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:31.257191 21214 hybrid_clock.cc:648] HybridClock initialized: now 1790639671257157 us; error 48 us; skew 500 ppm
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:31.260646 21214 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:31.261862 21214 webserver.cc:460] Webserver started at http://127.20.88.3:45057/ using document root <none> and password file <none>
I20260928 23:54:31.262537 21214 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:31.262643 21214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:31.262976 21214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:31.265724 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "6f6ee73fa9ce41ba9cc7d32e7758a9e3"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "f7eb7c6844a26e4773a3f09ae32bd434"
server_key_iv: "eb09fa46c1406a65929111006b850a81"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.266378 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "6f6ee73fa9ce41ba9cc7d32e7758a9e3"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "f7eb7c6844a26e4773a3f09ae32bd434"
server_key_iv: "eb09fa46c1406a65929111006b850a81"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.270938 21214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:31.273976 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.275493 21214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:31.275665 21214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "6f6ee73fa9ce41ba9cc7d32e7758a9e3"
format_stamp: "Formatted at 2026-09-28 23:54:31 on dist-test-slave-wvpw"
server_key: "f7eb7c6844a26e4773a3f09ae32bd434"
server_key_iv: "eb09fa46c1406a65929111006b850a81"
server_key_version: "encryptionkey@0"
I20260928 23:54:31.275808 21214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:31.291343 21214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:31.295554 21214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:31.295822 21214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:31.296609 21214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:31.297629 21214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:31.297807 21214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.297930 21214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:31.297981 21214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:31.308111 21214 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:45453
I20260928 23:54:31.308140 21342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:45453 every 8 connection(s)
I20260928 23:54:31.309603 21214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:31.319165 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21214
I20260928 23:54:31.319307 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:31.319631 20832 external_mini_cluster.cc:1428] Setting key ddc156426e88446d5989dab0c901fe1e
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:31.326468 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.311877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:43405 (local address 127.20.88.62:36961)
0928 23:54:31.312058 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:31.312064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:31.313150 (+ 1086us) server_negotiation.cc:407] Connection header received
0928 23:54:31.314631 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:31.314635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:31.314692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:31.314778 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:31.316205 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.316859 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.318187 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.318416 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.318509 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.322191 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:31.322197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:31.322205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:31.322235 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:31.323955 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.324649 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.324653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.324655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.324751 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.325190 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.325193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.325195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.325407 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:31.325531 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:31.326135 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 23:54:31.326277 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:54:31.327448 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.312391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36961 (local address 127.20.88.3:43405)
0928 23:54:31.312960 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:31.313011 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:31.314331 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:31.314977 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:31.314986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:31.315439 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:31.315991 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:31.316009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.317050 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.317053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:31.318044 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:31.318055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.318578 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.318580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:31.318746 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.319826 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:31.319868 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:31.321989 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:31.324103 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:31.324110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:31.324130 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:31.324528 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:31.324857 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:31.324863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:31.324866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:31.325075 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:31.325532 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:31.325541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:31.325857 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 23:54:31.326075 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 23:54:31.326460 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":148,"threads_started":1}
I20260928 23:54:31.329116 21343 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36961
I20260928 23:54:31.329447 21343 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:31.329988 21343 heartbeater.cc:511] Master 127.20.88.62:36961 requested a full tablet report, sending...
I20260928 23:54:31.331139 20884 ts_manager.cc:194] Registered new tserver with Master: 6f6ee73fa9ce41ba9cc7d32e7758a9e3 (127.20.88.3:45453)
I20260928 23:54:31.331846 20884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:43405
I20260928 23:54:31.335677 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:31.365479 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.354761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.20.88.1:39811)
0928 23:54:31.355122 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:31.355129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:31.355157 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:54:31.355363 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:31.355370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:31.355591 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:31.355722 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:31.356480 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.357259 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.358182 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.358591 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.358689 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.361357 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:31.361369 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:31.361384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:31.361426 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:31.363282 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.363753 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.363759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.363760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.363844 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.364201 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.364204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.364205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.364567 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:31.364739 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:31.365032 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 23:54:31.365228 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":84,"threads_started":1}
W20260928 23:54:31.366667 20989 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41898
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[20839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:31.394445 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:31.385076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.20.88.1:39811)
0928 23:54:31.385278 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:31.385282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:31.385341 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:54:31.385469 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:31.385472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:31.385516 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:31.385597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:31.386373 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.386936 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.387846 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:31.388118 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:31.388242 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:31.390999 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:31.391003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:31.391005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:31.391034 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:31.392672 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.393153 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.393161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.393163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.393242 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:31.393531 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:31.393535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:31.393537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:31.393727 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:31.393874 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:31.394134 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 23:54:31.394265 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 23:54:31.395625 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 20946
I20260928 23:54:31.401973 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21080
I20260928 23:54:31.409886 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21214
I20260928 23:54:31.416173 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 20852
2026-09-28T23:54:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2306 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[21357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[21357](info): set up 2 sockets
Sep 28 23:54:31 dist-test-slave-wvpw krb5kdc[21357](info): commencing operation
krb5kdc: starting...
W20260928 23:54:32.484540 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:32 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:32.616474 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:40733
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:40733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:32.732259 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:32.732581 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:32.732628 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:32.736779 21370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:32.736862 21370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:32.736883 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:32.736899 21370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:32.736915 21370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:32.742573 21370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:40733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:40733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:32.743736 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:32.744843 21370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:32.754657 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:32.754740 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:32.754868 21370 server_base.cc:1034] running on GCE node
W20260928 23:54:32.754722 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:32.755661 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:32.757042 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:32.758265 21370 hybrid_clock.cc:648] HybridClock initialized: now 1790639672758225 us; error 52 us; skew 500 ppm
Sep 28 23:54:32 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639672, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:32.761608 21370 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:32.762638 21370 webserver.cc:460] Webserver started at http://127.20.88.62:39897/ using document root <none> and password file <none>
I20260928 23:54:32.763274 21370 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:32.763383 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:32.763638 21370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:32.766017 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "7c7f7be2562340fda034a62de8714b2f"
format_stamp: "Formatted at 2026-09-28 23:54:32 on dist-test-slave-wvpw"
server_key: "6580ae107e412d7ea1f344d52b484db5"
server_key_iv: "64e57ca868432cac412c8df84dea0268"
server_key_version: "encryptionkey@0"
I20260928 23:54:32.766695 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "7c7f7be2562340fda034a62de8714b2f"
format_stamp: "Formatted at 2026-09-28 23:54:32 on dist-test-slave-wvpw"
server_key: "6580ae107e412d7ea1f344d52b484db5"
server_key_iv: "64e57ca868432cac412c8df84dea0268"
server_key_version: "encryptionkey@0"
I20260928 23:54:32.770985 21370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:32.773960 21385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:32.775532 21370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:32.775729 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "7c7f7be2562340fda034a62de8714b2f"
format_stamp: "Formatted at 2026-09-28 23:54:32 on dist-test-slave-wvpw"
server_key: "6580ae107e412d7ea1f344d52b484db5"
server_key_iv: "64e57ca868432cac412c8df84dea0268"
server_key_version: "encryptionkey@0"
I20260928 23:54:32.775884 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:32.795288 21370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:32.799321 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:32.799574 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:32.807459 21370 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:40733
I20260928 23:54:32.807485 21437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:40733 every 8 connection(s)
I20260928 23:54:32.808573 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:32.811538 21438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:32.813700 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21370
I20260928 23:54:32.813802 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:32.814088 20832 external_mini_cluster.cc:1428] Setting key 4faa843a546b07548bd96eff0162679f
I20260928 23:54:32.818341 21438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: Bootstrap starting.
I20260928 23:54:32.820924 21438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:32.821770 21438 log.cc:824] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:54:32 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:32.823840 21438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: No bootstrap required, opened a new log
I20260928 23:54:32.827383 21438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } }
I20260928 23:54:32.827684 21438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:32.827766 21438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7f7be2562340fda034a62de8714b2f, State: Initialized, Role: FOLLOWER
I20260928 23:54:32.828332 21438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } }
I20260928 23:54:32.828519 21438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:32.828600 21438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:32.828725 21438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:32.829903 21438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } }
I20260928 23:54:32.830286 21438 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7f7be2562340fda034a62de8714b2f; no voters:
I20260928 23:54:32.830477 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:32.815754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.20.88.62:40733)
0928 23:54:32.816333 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:32.816351 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:32.816416 (+ 65us) server_negotiation.cc:407] Connection header received
0928 23:54:32.817323 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:32.817347 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:32.817782 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:32.818170 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:32.819032 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:32.820062 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:32.821033 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:32.821628 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:32.821822 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:32.824500 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:32.824506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:32.824523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:32.824554 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:32.826962 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:32.827493 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:32.827499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:32.827504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:32.827631 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:32.827993 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:32.828001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:32.828002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:32.828608 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:32.828854 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:32.829177 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 23:54:32.829518 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":155,"threads_started":1}
I20260928 23:54:32.830689 21438 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:32.830807 21443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:32.830984 21443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [term 1 LEADER]: Becoming Leader. State: Replica: 7c7f7be2562340fda034a62de8714b2f, State: Running, Role: LEADER
I20260928 23:54:32.831390 21443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } }
I20260928 23:54:32.832103 21438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:32.834323 21444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7f7be2562340fda034a62de8714b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } } }
I20260928 23:54:32.834471 21444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:32.834868 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7f7be2562340fda034a62de8714b2f. Latest consensus state: current_term: 1 leader_uuid: "7c7f7be2562340fda034a62de8714b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f7be2562340fda034a62de8714b2f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40733 } } }
I20260928 23:54:32.835189 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f [sys.catalog]: This master's current role is: LEADER
W20260928 23:54:32.837376 21459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:32.837447 21459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:32.837533 21458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:32.838300 21458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:32.844144 21458 catalog_manager.cc:1347] Generated new cluster ID: b53e76357e6d4cfaabcae8fc28313280
I20260928 23:54:32.844276 21458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:32.854225 21458 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:32.855479 21458 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:32.863209 21458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7f7be2562340fda034a62de8714b2f: Generated new TSK 0
I20260928 23:54:32.864002 21458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:32.899619 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:33.023762 21464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:33.024091 21464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:33.024138 21464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:33.028683 21464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:33.028759 21464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:33.028821 21464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:33.034654 21464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:33.035846 21464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:33.037002 21464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:33.047102 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:33.047102 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:33.047124 21469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:33.047485 21464 server_base.cc:1034] running on GCE node
I20260928 23:54:33.047989 21464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:33.048764 21464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:33.049945 21464 hybrid_clock.cc:648] HybridClock initialized: now 1790639673049927 us; error 33 us; skew 500 ppm
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:33.053100 21464 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:33.054024 21464 webserver.cc:460] Webserver started at http://127.20.88.1:45521/ using document root <none> and password file <none>
I20260928 23:54:33.054576 21464 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:33.054636 21464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:33.054804 21464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:33.057083 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "d12f063e201e4ef9ae81068e3afb1880"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "514e11f6d796f95d51a4a039e20bbdda"
server_key_iv: "ff237759f13e21a505ff4ef836e486db"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.057659 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "d12f063e201e4ef9ae81068e3afb1880"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "514e11f6d796f95d51a4a039e20bbdda"
server_key_iv: "ff237759f13e21a505ff4ef836e486db"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.061561 21464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:54:33.064210 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.065762 21464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:54:33.065948 21464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "d12f063e201e4ef9ae81068e3afb1880"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "514e11f6d796f95d51a4a039e20bbdda"
server_key_iv: "ff237759f13e21a505ff4ef836e486db"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.066092 21464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:33.082278 21464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:33.085773 21464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:33.086019 21464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:33.086701 21464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:33.087648 21464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:33.087729 21464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.087809 21464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:33.087858 21464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.098001 21464 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39225
I20260928 23:54:33.098018 21592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39225 every 8 connection(s)
I20260928 23:54:33.099045 21464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:33.107578 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21464
I20260928 23:54:33.107685 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:33.107923 20832 external_mini_cluster.cc:1428] Setting key 7b643bdcfdbcd3777b8e8a13c82197f0
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:33.115074 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.101223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:48317 (local address 127.20.88.62:40733)
0928 23:54:33.101384 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.101389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.102365 (+ 976us) server_negotiation.cc:407] Connection header received
0928 23:54:33.103386 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.103389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.103449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.103556 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.104895 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.105460 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.106674 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.106918 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.107036 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.110400 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.110404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.110406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.110432 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.112442 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.113038 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.113045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.113050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.113147 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.113521 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.113527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.113532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.113808 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.113918 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.114620 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.114801 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:54:33.115926 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.101681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:40733 (local address 127.20.88.1:48317)
0928 23:54:33.102192 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.102239 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.103147 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.103745 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.103752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.104139 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:33.104734 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.104749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.105604 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.105607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.106516 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.106526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.107085 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.107089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.107303 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.108049 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:33.108068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:33.110141 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:33.112579 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.112586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.112602 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.112919 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.113251 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.113253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.113255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.113420 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.113955 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:33.113960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:33.114338 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.114529 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.114828 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260928 23:54:33.117420 21593 heartbeater.cc:347] Connected to a master server at 127.20.88.62:40733
I20260928 23:54:33.117761 21593 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:54:33.118455 21593 heartbeater.cc:511] Master 127.20.88.62:40733 requested a full tablet report, sending...
I20260928 23:54:33.120407 21402 ts_manager.cc:194] Registered new tserver with Master: d12f063e201e4ef9ae81068e3afb1880 (127.20.88.1:39225)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:33.121775 21402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:48317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:33.140067 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:33.257221 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:33.257597 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:33.257705 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:33.262171 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:33.262276 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:33.262382 21598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:33.268091 21598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:33.269345 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:33.270504 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:33.280782 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:33.281056 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:33.281319 21598 server_base.cc:1034] running on GCE node
W20260928 23:54:33.280834 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:33.281791 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:33.282496 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:33.283699 21598 hybrid_clock.cc:648] HybridClock initialized: now 1790639673283655 us; error 56 us; skew 500 ppm
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:33.286604 21598 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:33.287533 21598 webserver.cc:460] Webserver started at http://127.20.88.2:45307/ using document root <none> and password file <none>
I20260928 23:54:33.288141 21598 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:33.288234 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:33.288517 21598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:33.290674 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "0e85105395b84bd094975b3e668d225a"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "a35565ab01ed99d1aedc8d55bc62e43b"
server_key_iv: "b10397894ff73959203e0a0bc3f9ec25"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.291255 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "0e85105395b84bd094975b3e668d225a"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "a35565ab01ed99d1aedc8d55bc62e43b"
server_key_iv: "b10397894ff73959203e0a0bc3f9ec25"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.295214 21598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 23:54:33.298269 21613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.299738 21598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:33.299912 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "0e85105395b84bd094975b3e668d225a"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "a35565ab01ed99d1aedc8d55bc62e43b"
server_key_iv: "b10397894ff73959203e0a0bc3f9ec25"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.300057 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:33.332571 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:33.336584 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:33.336822 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:33.337513 21598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:33.338507 21598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:33.338594 21598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.338692 21598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:33.338748 21598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.348311 21598 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:38259
I20260928 23:54:33.348327 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:38259 every 8 connection(s)
I20260928 23:54:33.349285 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:33.357620 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21598
I20260928 23:54:33.357750 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:33.358144 20832 external_mini_cluster.cc:1428] Setting key 897f4f812bc7b3fb84f6a77f9648ce11
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:33.364329 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.351270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:33779 (local address 127.20.88.62:40733)
0928 23:54:33.351421 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.351425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.352643 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 23:54:33.353485 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.353488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.353534 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.353609 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.354912 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.355459 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.356729 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.357060 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.357144 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.360276 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.360282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.360289 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.360324 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.361885 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.362543 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.362550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.362554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.362647 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.363100 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.363104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.363106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.363331 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.363459 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.363986 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.364116 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:54:33.365272 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.351706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:40733 (local address 127.20.88.2:33779)
0928 23:54:33.352477 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.352522 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.353288 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.353783 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.353791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.354193 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:33.354726 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.354741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.355584 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.355586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.356615 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.356625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.357197 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.357200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.357403 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.358188 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:33.358242 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:33.360064 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:33.362002 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.362010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.362030 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.362413 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.362747 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.362754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.362759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.363002 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.363448 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:33.363456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:33.363735 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.363935 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.364318 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":115,"threads_started":1}
I20260928 23:54:33.366570 21727 heartbeater.cc:347] Connected to a master server at 127.20.88.62:40733
I20260928 23:54:33.366855 21727 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:33.367518 21727 heartbeater.cc:511] Master 127.20.88.62:40733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:54:33.368801 21402 ts_manager.cc:194] Registered new tserver with Master: 0e85105395b84bd094975b3e668d225a (127.20.88.2:38259)
I20260928 23:54:33.369360 21402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:33779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:33.387771 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:33.508342 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:33.508680 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:33.508730 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:33.513106 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:33.513204 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:33.513270 21732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:33.519140 21732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:40733
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:33.520486 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:33.521643 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:33.532437 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:33.532437 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:33.532497 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:33.532989 21732 server_base.cc:1034] running on GCE node
I20260928 23:54:33.533545 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:33.534339 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:33.535542 21732 hybrid_clock.cc:648] HybridClock initialized: now 1790639673535518 us; error 37 us; skew 500 ppm
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:33.538769 21732 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:33.539812 21732 webserver.cc:460] Webserver started at http://127.20.88.3:46389/ using document root <none> and password file <none>
I20260928 23:54:33.540522 21732 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:33.540597 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:33.540777 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:33.543004 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "a82ae88df6b6421cb5bd30631dbd2565"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "cc7624923243deb8f1036bf21c3eca8a"
server_key_iv: "43acc8c4dfd0af45c4f374637a878743"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.543574 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "a82ae88df6b6421cb5bd30631dbd2565"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "cc7624923243deb8f1036bf21c3eca8a"
server_key_iv: "43acc8c4dfd0af45c4f374637a878743"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.548071 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:54:33.550957 21747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.552556 21732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:33.552724 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "a82ae88df6b6421cb5bd30631dbd2565"
format_stamp: "Formatted at 2026-09-28 23:54:33 on dist-test-slave-wvpw"
server_key: "cc7624923243deb8f1036bf21c3eca8a"
server_key_iv: "43acc8c4dfd0af45c4f374637a878743"
server_key_version: "encryptionkey@0"
I20260928 23:54:33.552860 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:33.573580 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:33.577359 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:33.577602 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:33.578255 21732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:33.579254 21732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:33.579326 21732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.579403 21732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:33.579459 21732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:33.588404 21732 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:44457
I20260928 23:54:33.588449 21860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:44457 every 8 connection(s)
I20260928 23:54:33.589394 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:33.595329 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21732
I20260928 23:54:33.595463 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:33.595739 20832 external_mini_cluster.cc:1428] Setting key e65c0eb81869f492db2941d83614e0a0
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:33.604651 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.591484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:58697 (local address 127.20.88.62:40733)
0928 23:54:33.591743 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.591747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.592627 (+ 880us) server_negotiation.cc:407] Connection header received
0928 23:54:33.593594 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.593597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.593658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.593767 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.595042 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.595838 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.597684 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.597925 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.598086 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.600797 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.600819 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.600821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.600851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.602305 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.602939 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.602942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.602943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.603033 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.603381 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.603383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.603385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.603580 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.603728 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.604396 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.604511 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260928 23:54:33.605580 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.591876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:40733 (local address 127.20.88.3:58697)
0928 23:54:33.592459 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.592508 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.593357 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.593969 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.593977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.594357 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:33.594876 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.594890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.595997 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.596000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.597512 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.597524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.598145 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.598149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.598351 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.599014 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:33.599032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:33.600593 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:33.602438 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.602444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.602459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.602756 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.603125 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:33.603127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:33.603128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:33.603281 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:33.603764 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:33.603770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:33.604102 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.604335 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.604628 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20260928 23:54:33.606747 21861 heartbeater.cc:347] Connected to a master server at 127.20.88.62:40733
I20260928 23:54:33.607031 21861 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:33.607577 21861 heartbeater.cc:511] Master 127.20.88.62:40733 requested a full tablet report, sending...
I20260928 23:54:33.608963 21402 ts_manager.cc:194] Registered new tserver with Master: a82ae88df6b6421cb5bd30631dbd2565 (127.20.88.3:44457)
I20260928 23:54:33.609712 21402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:58697
I20260928 23:54:33.610649 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:54:33.622742 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.614557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.20.88.62:40733)
0928 23:54:33.614716 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.614720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.614864 (+ 144us) server_negotiation.cc:407] Connection header received
0928 23:54:33.614974 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.614976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.615019 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.615082 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.615834 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.616508 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.617417 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.617658 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.617741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.619374 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.619378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.619381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.619407 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.621022 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.621452 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.621455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.621456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.621536 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.621816 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.621819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.621820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.622113 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.622231 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.622504 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.622614 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:54:33.628441 21402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:33.631633 21402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:33.647516 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.20.88.1:39225)
0928 23:54:33.642370 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.642375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.642643 (+ 268us) server_negotiation.cc:407] Connection header received
0928 23:54:33.642764 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.642770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.642990 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.643140 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:33.643443 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644690 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.646202 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.646792 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.646935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.647100 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.647257 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.647363 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":88,"threads_started":1}
I20260928 23:54:33.647504 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39225 (local address 127.0.0.1:47570)
0928 23:54:33.642536 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.642549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.642652 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.643137 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.643142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.643165 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:33.643306 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.643319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644831 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.644837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.646079 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.646089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.646927 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.646931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.647041 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.647078 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.647215 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.647349 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":93,"thread_start_us":113,"threads_started":1}
I20260928 23:54:33.648722 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:44457 (local address 127.0.0.1:50802)
0928 23:54:33.642349 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.642403 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.642545 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.643304 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.643307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.643334 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:33.643494 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.643502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644724 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.644729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.647209 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.647221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.648300 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.648304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.648394 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.648432 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.648516 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.648590 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":122,"thread_start_us":94,"threads_started":1}
I20260928 23:54:33.649046 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.20.88.3:44457)
0928 23:54:33.642086 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.642098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.642505 (+ 407us) server_negotiation.cc:407] Connection header received
0928 23:54:33.642765 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.642770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.642989 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.643141 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:33.643623 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644590 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.647348 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.648159 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.648307 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.648464 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.648775 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.648890 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":160,"threads_started":1}
I20260928 23:54:33.650803 21527 tablet_service.cc:1467] Processing CreateTablet for tablet 3179905c58364dc7af5e3f9d257fdbdd (DEFAULT_TABLE table=test-table [id=843b71a4fe2443fe879bf047876d09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:33.652005 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38259 (local address 127.0.0.1:54360)
0928 23:54:33.642815 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.642824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.642915 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.643401 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.643404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.643417 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:33.643515 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.643521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644865 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.644869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.646087 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.646116 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.647702 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.647706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.651578 (+ 3872us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.651603 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.651719 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.651808 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":120,"spinlock_wait_cycles":1280,"thread_start_us":101,"threads_started":1}
I20260928 23:54:33.652019 21527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3179905c58364dc7af5e3f9d257fdbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:33.652022 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.641718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.20.88.2:38259)
0928 23:54:33.642184 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.642197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.642908 (+ 711us) server_negotiation.cc:407] Connection header received
0928 23:54:33.643052 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.643057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.643261 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.643418 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:33.643589 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.644734 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.646256 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.647111 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.647273 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.647431 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.651763 (+ 4332us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.651880 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":163,"threads_started":1}
I20260928 23:54:33.651717 21795 tablet_service.cc:1467] Processing CreateTablet for tablet 3179905c58364dc7af5e3f9d257fdbdd (DEFAULT_TABLE table=test-table [id=843b71a4fe2443fe879bf047876d09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:33.652814 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3179905c58364dc7af5e3f9d257fdbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:33.654860 21661 tablet_service.cc:1467] Processing CreateTablet for tablet 3179905c58364dc7af5e3f9d257fdbdd (DEFAULT_TABLE table=test-table [id=843b71a4fe2443fe879bf047876d09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:33.655853 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3179905c58364dc7af5e3f9d257fdbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:33.659948 21878 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Bootstrap starting.
I20260928 23:54:33.661382 21879 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Bootstrap starting.
I20260928 23:54:33.662568 21878 tablet_bootstrap.cc:654] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:33.663620 21879 tablet_bootstrap.cc:654] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:33.663681 21880 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Bootstrap starting.
I20260928 23:54:33.663901 21878 log.cc:824] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:33.664901 21879 log.cc:824] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:33.665933 21880 tablet_bootstrap.cc:654] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:33.666363 21878 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: No bootstrap required, opened a new log
I20260928 23:54:33.666594 21878 ts_tablet_manager.cc:1397] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260928 23:54:33.667083 21879 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: No bootstrap required, opened a new log
I20260928 23:54:33.667083 21880 log.cc:824] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:33.667259 21879 ts_tablet_manager.cc:1397] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:33.668859 21880 tablet_bootstrap.cc:492] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: No bootstrap required, opened a new log
I20260928 23:54:33.669052 21880 ts_tablet_manager.cc:1397] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:54:33.671402 21878 raft_consensus.cc:348] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.671716 21878 raft_consensus.cc:374] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:33.671787 21878 raft_consensus.cc:729] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12f063e201e4ef9ae81068e3afb1880, State: Initialized, Role: FOLLOWER
I20260928 23:54:33.671731 21879 raft_consensus.cc:348] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.671994 21879 raft_consensus.cc:374] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:33.672055 21879 raft_consensus.cc:729] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82ae88df6b6421cb5bd30631dbd2565, State: Initialized, Role: FOLLOWER
I20260928 23:54:33.672394 21878 consensus_queue.cc:260] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.672458 21879 consensus_queue.cc:260] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.672670 21880 raft_consensus.cc:348] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.672966 21880 raft_consensus.cc:374] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:33.673029 21880 raft_consensus.cc:729] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e85105395b84bd094975b3e668d225a, State: Initialized, Role: FOLLOWER
I20260928 23:54:33.673439 21861 heartbeater.cc:503] Master 127.20.88.62:40733 was elected leader, sending a full tablet report...
I20260928 23:54:33.673498 21879 ts_tablet_manager.cc:1428] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:54:33.673543 21880 consensus_queue.cc:260] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.674247 21727 heartbeater.cc:503] Master 127.20.88.62:40733 was elected leader, sending a full tablet report...
I20260928 23:54:33.674504 21593 heartbeater.cc:503] Master 127.20.88.62:40733 was elected leader, sending a full tablet report...
I20260928 23:54:33.674595 21880 ts_tablet_manager.cc:1428] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:54:33.675580 21878 ts_tablet_manager.cc:1428] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
W20260928 23:54:33.840521 21862 tablet.cc:2369] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:33.842290 21886 raft_consensus.cc:482] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:33.842541 21886 raft_consensus.cc:504] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.843858 21886 leader_election.cc:288] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a82ae88df6b6421cb5bd30631dbd2565 (127.20.88.3:44457), d12f063e201e4ef9ae81068e3afb1880 (127.20.88.1:39225)
I20260928 23:54:33.847889 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.844482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:60895 (local address 127.20.88.1:39225)
0928 23:54:33.844616 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.844622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.844640 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:54:33.844696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.844700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.844756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.844839 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:33.845224 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.845750 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.846844 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.847262 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.847400 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.847474 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.847668 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.847747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:54:33.847918 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.844154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39225 (local address 127.20.88.2:60895)
0928 23:54:33.844381 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.844409 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.844514 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.844922 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.844928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.844947 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:33.845101 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.845108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.845879 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.845883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.846709 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.846718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.847397 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.847401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.847542 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.847559 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.847651 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.847733 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":71,"spinlock_wait_cycles":25856}
I20260928 23:54:33.847919 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.844320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:54625 (local address 127.20.88.3:44457)
0928 23:54:33.844469 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.844475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.844688 (+ 213us) server_negotiation.cc:407] Connection header received
0928 23:54:33.844797 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.844800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.844843 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.844939 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:33.845251 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.845750 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.846908 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.847314 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.847455 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.847522 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.847722 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.847793 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:54:33.847946 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.844174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:44457 (local address 127.20.88.2:54625)
0928 23:54:33.844562 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:33.844575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:33.844695 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:33.844964 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:33.844966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:33.844995 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:33.845137 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.845142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.845876 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.845879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.846775 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:33.846784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.847470 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.847474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:33.847600 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.847618 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:54:33.847706 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:54:33.847778 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":39,"thread_start_us":128,"threads_started":1}
I20260928 23:54:33.848646 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3179905c58364dc7af5e3f9d257fdbdd" candidate_uuid: "0e85105395b84bd094975b3e668d225a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12f063e201e4ef9ae81068e3afb1880" is_pre_election: true
I20260928 23:54:33.848645 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3179905c58364dc7af5e3f9d257fdbdd" candidate_uuid: "0e85105395b84bd094975b3e668d225a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82ae88df6b6421cb5bd30631dbd2565" is_pre_election: true
I20260928 23:54:33.848968 21815 raft_consensus.cc:2454] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e85105395b84bd094975b3e668d225a in term 0.
I20260928 23:54:33.848973 21547 raft_consensus.cc:2454] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e85105395b84bd094975b3e668d225a in term 0.
I20260928 23:54:33.849499 21616 leader_election.cc:302] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e85105395b84bd094975b3e668d225a, a82ae88df6b6421cb5bd30631dbd2565; no voters:
I20260928 23:54:33.849795 21886 raft_consensus.cc:2781] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:33.849939 21886 raft_consensus.cc:482] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:33.850011 21886 raft_consensus.cc:3037] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:54:33.850426 21728 tablet.cc:2369] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:54:33.850458 21594 tablet.cc:2369] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:33.851306 21886 raft_consensus.cc:504] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.851687 21886 leader_election.cc:288] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [CANDIDATE]: Term 1 election: Requested vote from peers a82ae88df6b6421cb5bd30631dbd2565 (127.20.88.3:44457), d12f063e201e4ef9ae81068e3afb1880 (127.20.88.1:39225)
I20260928 23:54:33.852176 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3179905c58364dc7af5e3f9d257fdbdd" candidate_uuid: "0e85105395b84bd094975b3e668d225a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82ae88df6b6421cb5bd30631dbd2565"
I20260928 23:54:33.852202 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3179905c58364dc7af5e3f9d257fdbdd" candidate_uuid: "0e85105395b84bd094975b3e668d225a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12f063e201e4ef9ae81068e3afb1880"
I20260928 23:54:33.852394 21547 raft_consensus.cc:3037] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:33.852396 21815 raft_consensus.cc:3037] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:33.853520 21547 raft_consensus.cc:2454] T 3179905c58364dc7af5e3f9d257fdbdd P d12f063e201e4ef9ae81068e3afb1880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e85105395b84bd094975b3e668d225a in term 1.
I20260928 23:54:33.853546 21815 raft_consensus.cc:2454] T 3179905c58364dc7af5e3f9d257fdbdd P a82ae88df6b6421cb5bd30631dbd2565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e85105395b84bd094975b3e668d225a in term 1.
I20260928 23:54:33.853879 21615 leader_election.cc:302] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e85105395b84bd094975b3e668d225a, d12f063e201e4ef9ae81068e3afb1880; no voters:
I20260928 23:54:33.854094 21886 raft_consensus.cc:2781] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:33.854353 21886 raft_consensus.cc:686] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [term 1 LEADER]: Becoming Leader. State: Replica: 0e85105395b84bd094975b3e668d225a, State: Running, Role: LEADER
I20260928 23:54:33.854750 21886 consensus_queue.cc:237] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } }
I20260928 23:54:33.858568 21402 catalog_manager.cc:5495] T 3179905c58364dc7af5e3f9d257fdbdd P 0e85105395b84bd094975b3e668d225a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e85105395b84bd094975b3e668d225a (127.20.88.2). New cstate: current_term: 1 leader_uuid: "0e85105395b84bd094975b3e668d225a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a82ae88df6b6421cb5bd30631dbd2565" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12f063e201e4ef9ae81068e3afb1880" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e85105395b84bd094975b3e668d225a" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38259 } health_report { overall_health: HEALTHY } } }
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:33.889557 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.879767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.20.88.1:39225)
0928 23:54:33.879977 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.879982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.880156 (+ 174us) server_negotiation.cc:407] Connection header received
0928 23:54:33.880335 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.880338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.880395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.880491 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.881220 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.881731 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.882602 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.882828 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.882950 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.885601 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.885615 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.885631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.885661 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.887420 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.887922 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.887927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.887928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.888014 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.888387 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.888390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.888392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.888668 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.888882 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.889208 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.889388 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:33.917987 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.908484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.20.88.1:39225)
0928 23:54:33.908664 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.908669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.908809 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:54:33.909029 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.909033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.909080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.909174 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.909883 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.910423 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.911378 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.911610 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.911726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.914488 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.914491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.914493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.914524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.916010 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.916592 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.916596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.916598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.916689 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.916996 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.917000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.917001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.917188 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.917328 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.917668 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.917805 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:54:33 dist-test-slave-wvpw krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:33.945901 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:33.936588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.20.88.1:39225)
0928 23:54:33.936725 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:33.936731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:33.936950 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:54:33.937123 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:33.937126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:33.937180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:33.937272 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:33.938013 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.938602 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.939495 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:33.939731 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:33.939846 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:33.942533 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:33.942536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:33.942538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:33.942569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:33.944035 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.944605 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.944609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.944611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.944701 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:33.945005 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:33.945008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:33.945010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:33.945202 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:33.945356 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:33.945593 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 23:54:33.945723 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:54:33.947417 21507 tablet_service.cc:2908] Rejecting scan request for tablet 3179905c58364dc7af5e3f9d257fdbdd: Uninitialized: clean time has not yet been initialized
I20260928 23:54:33.949735 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21464
I20260928 23:54:33.955955 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21598
I20260928 23:54:33.961670 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21732
I20260928 23:54:33.967404 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21370
2026-09-28T23:54:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2551 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:34 dist-test-slave-wvpw krb5kdc[21902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:34 dist-test-slave-wvpw krb5kdc[21902](info): set up 2 sockets
Sep 28 23:54:34 dist-test-slave-wvpw krb5kdc[21902](info): commencing operation
krb5kdc: starting...
W20260928 23:54:35.048421 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:35.179087 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32951
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:32951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:35.297202 21915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:35.297616 21915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:35.297704 21915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:35.301997 21915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:35.302125 21915 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:54:35.302156 21915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:35.302203 21915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:35.302240 21915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:35.302271 21915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:35.308059 21915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:32951
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:35.309379 21915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:35.310645 21915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:35.321071 21921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:35.321044 21920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:35.321194 21915 server_base.cc:1034] running on GCE node
W20260928 23:54:35.321029 21923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:35.322052 21915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:35.323328 21915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:35.324546 21915 hybrid_clock.cc:648] HybridClock initialized: now 1790639675324511 us; error 51 us; skew 500 ppm
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:35.327628 21915 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:35.328754 21915 webserver.cc:460] Webserver started at http://127.20.88.62:45231/ using document root <none> and password file <none>
I20260928 23:54:35.329303 21915 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:35.329373 21915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:35.329546 21915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:35.331648 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "87849a41a94b4b8abec84565f2e5572e"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "4c4797086cfa25320f929fa466609356"
server_key_iv: "30e2527a3aae2dcbf6729556b0b569da"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.332172 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "87849a41a94b4b8abec84565f2e5572e"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "4c4797086cfa25320f929fa466609356"
server_key_iv: "30e2527a3aae2dcbf6729556b0b569da"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.336069 21915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:54:35.338712 21930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:35.340126 21915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:35.340368 21915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "87849a41a94b4b8abec84565f2e5572e"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "4c4797086cfa25320f929fa466609356"
server_key_iv: "30e2527a3aae2dcbf6729556b0b569da"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.340508 21915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:35.356417 21915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:35.360177 21915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:35.360448 21915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:35.368955 21915 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:32951
I20260928 23:54:35.368965 21992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:32951 every 8 connection(s)
I20260928 23:54:35.370004 21915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:35.372965 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:35.376550 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 21915
I20260928 23:54:35.376693 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:35.376958 20832 external_mini_cluster.cc:1428] Setting key 666dbd2246d00f1825b8b58e4c4ab97c
I20260928 23:54:35.379546 21993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Bootstrap starting.
I20260928 23:54:35.381965 21993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:35.382754 21993 log.cc:824] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:35.384900 21993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: No bootstrap required, opened a new log
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:35.388442 21993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } }
I20260928 23:54:35.388736 21993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:35.388818 21993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87849a41a94b4b8abec84565f2e5572e, State: Initialized, Role: FOLLOWER
I20260928 23:54:35.389339 21993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } }
I20260928 23:54:35.389503 21993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:35.389609 21993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:35.389714 21993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:35.390681 21993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } }
I20260928 23:54:35.391069 21993 leader_election.cc:302] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87849a41a94b4b8abec84565f2e5572e; no voters:
I20260928 23:54:35.391458 21993 leader_election.cc:288] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:35.391578 21998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:35.391808 21998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [term 1 LEADER]: Becoming Leader. State: Replica: 87849a41a94b4b8abec84565f2e5572e, State: Running, Role: LEADER
I20260928 23:54:35.392148 21998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } }
I20260928 23:54:35.392568 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.378595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.20.88.62:32951)
0928 23:54:35.379168 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:35.379185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:35.379226 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:54:35.379979 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:35.380001 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:35.380446 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:35.380804 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:35.381656 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.382743 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.383704 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.384200 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.384410 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.386937 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:35.386942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:35.386957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:35.386986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:35.388967 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.389441 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.389447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.389453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.389657 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.389973 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.389977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.389979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.390462 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:35.390678 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:35.390997 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 23:54:35.391354 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":182,"threads_started":1}
I20260928 23:54:35.393098 21993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:35.393463 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87849a41a94b4b8abec84565f2e5572e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } } }
I20260928 23:54:35.393584 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:35.394207 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87849a41a94b4b8abec84565f2e5572e. Latest consensus state: current_term: 1 leader_uuid: "87849a41a94b4b8abec84565f2e5572e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87849a41a94b4b8abec84565f2e5572e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32951 } } }
I20260928 23:54:35.394306 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:35.394903 22002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:35.398753 22002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:35.405987 22002 catalog_manager.cc:1347] Generated new cluster ID: c90a5bdf07bd45a6a4c4b5c3a30b7893
I20260928 23:54:35.406101 22002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:35.418846 22002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:35.420298 22002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:35.428144 22002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Generated new TSK 0
I20260928 23:54:35.429123 22002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:35.468801 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32951
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:35.588006 22019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:35.588410 22019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:35.588507 22019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:35.592792 22019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:35.592883 22019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:35.592973 22019 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:35.593057 22019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:35.598968 22019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:35.600186 22019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:35.601379 22019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:35.612902 22027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:35.612953 22025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:35.612907 22024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:35.613111 22019 server_base.cc:1034] running on GCE node
I20260928 23:54:35.613657 22019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:35.614408 22019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:35.615645 22019 hybrid_clock.cc:648] HybridClock initialized: now 1790639675615603 us; error 54 us; skew 500 ppm
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:35.625605 22019 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:35.626729 22019 webserver.cc:460] Webserver started at http://127.20.88.1:32913/ using document root <none> and password file <none>
I20260928 23:54:35.627406 22019 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:35.627511 22019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:35.627763 22019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:35.630098 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "7a5004b793d24050a1dc6d45889aa4ac"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "6ceb75cd6173209240ffbb737202184f"
server_key_iv: "adb3a8c974fb1b4c359d773d37bb4c2f"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.630733 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "7a5004b793d24050a1dc6d45889aa4ac"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "6ceb75cd6173209240ffbb737202184f"
server_key_iv: "adb3a8c974fb1b4c359d773d37bb4c2f"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.635028 22019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:54:35.638039 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:35.639487 22019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:35.639673 22019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "7a5004b793d24050a1dc6d45889aa4ac"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "6ceb75cd6173209240ffbb737202184f"
server_key_iv: "adb3a8c974fb1b4c359d773d37bb4c2f"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.639820 22019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:35.652611 22019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:35.656352 22019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:35.656574 22019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:35.658264 22019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:35.658340 22019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:35.658413 22019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:35.658455 22019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:35.669786 22019 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:42445
I20260928 23:54:35.669857 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:42445 every 8 connection(s)
I20260928 23:54:35.670948 22019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:35.673714 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.659192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:49859 (local address 127.20.88.62:32951)
0928 23:54:35.659356 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:35.659359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:35.660506 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 23:54:35.661705 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:35.661709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:35.661769 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:35.661826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:35.663510 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.664178 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.665796 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.666095 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.666230 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.669686 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:35.669689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:35.669691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:35.669718 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:35.671417 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.672049 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.672052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.672054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.672138 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.672573 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.672576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.672578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.672783 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:35.672881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:35.673434 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 23:54:35.673574 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:54:35.674506 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.659716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.1:49859)
0928 23:54:35.660344 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:35.660393 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:35.661412 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:35.662225 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:35.662232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:35.662701 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:35.663346 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.663362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.664359 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.664365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.665657 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.665671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.666299 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.666304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.666522 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.667742 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:35.667759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:35.669475 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:35.671577 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.671583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.671606 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.671942 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.672275 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.672279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.672281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.672441 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.672908 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:35.672914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:35.673161 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 23:54:35.673326 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 23:54:35.673591 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":129,"threads_started":1}
I20260928 23:54:35.675911 22150 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32951
I20260928 23:54:35.676059 22150 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:35.676192 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22019
I20260928 23:54:35.676358 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:35.676632 20832 external_mini_cluster.cc:1428] Setting key 46c15fe74b590ab86ad5915958283265
I20260928 23:54:35.676681 22150 heartbeater.cc:511] Master 127.20.88.62:32951 requested a full tablet report, sending...
I20260928 23:54:35.678236 21943 ts_manager.cc:194] Registered new tserver with Master: 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445)
I20260928 23:54:35.679900 21943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:49859
I20260928 23:54:35.686271 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.678603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:46083 (local address 127.20.88.62:32951)
0928 23:54:35.678896 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:35.678903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:35.678950 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:54:35.679133 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:35.679136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:35.679183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:35.679265 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:35.679706 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.680459 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.681436 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.681711 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.681833 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.683043 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:35.683054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:35.683056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:35.683078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:35.684637 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.685069 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.685072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.685075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.685155 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.685442 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.685446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.685448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.685621 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:35.685726 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:35.685985 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 23:54:35.686102 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260928 23:54:35.686302 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.678568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.1:46083)
0928 23:54:35.678870 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:35.678885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:35.679018 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:35.679281 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:35.679284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:35.679494 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:35.679594 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.679599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.680608 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.680612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.681307 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.681320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.681838 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.681841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.681936 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.682541 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:35.682553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:35.682895 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:35.684758 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.684762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.684764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.684969 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.685255 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.685258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.685260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.685353 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.685725 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:35.685728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:35.685825 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 23:54:35.685930 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:54:35.686089 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":4,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:35.707962 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32951
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:35.825147 22159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:35.825452 22159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:35.825500 22159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:35.829751 22159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:35.829823 22159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:35.829865 22159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:35.829900 22159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:35.835592 22159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:35.836798 22159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:35.837859 22159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:35.848654 22164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:35.848711 22159 server_base.cc:1034] running on GCE node
W20260928 23:54:35.849004 22167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:35.849013 22165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:35.849581 22159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:35.850299 22159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:35.851509 22159 hybrid_clock.cc:648] HybridClock initialized: now 1790639675851485 us; error 37 us; skew 500 ppm
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:35.854646 22159 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:35.855600 22159 webserver.cc:460] Webserver started at http://127.20.88.2:33587/ using document root <none> and password file <none>
I20260928 23:54:35.856173 22159 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:35.856304 22159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:35.856614 22159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:35.858840 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "d8d8fa3b59e24aa5b73a9c53093ae047"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "64a9ef18c08a5d9e8eae3e3aea153547"
server_key_iv: "95d254c85101655202c22cb6a67f04e4"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.859467 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d8fa3b59e24aa5b73a9c53093ae047"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "64a9ef18c08a5d9e8eae3e3aea153547"
server_key_iv: "95d254c85101655202c22cb6a67f04e4"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.863888 22159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:35.866727 22174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:35.868115 22159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:35.868295 22159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "d8d8fa3b59e24aa5b73a9c53093ae047"
format_stamp: "Formatted at 2026-09-28 23:54:35 on dist-test-slave-wvpw"
server_key: "64a9ef18c08a5d9e8eae3e3aea153547"
server_key_iv: "95d254c85101655202c22cb6a67f04e4"
server_key_version: "encryptionkey@0"
I20260928 23:54:35.868434 22159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:35.887785 22159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:35.891233 22159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:35.891467 22159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:35.893155 22159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:35.893206 22159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:35.893249 22159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:35.893306 22159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:54:35 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:35.904758 22159 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:34063
I20260928 23:54:35.904839 22289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:34063 every 8 connection(s)
I20260928 23:54:35.905978 22159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:35.908999 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.894119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:53441 (local address 127.20.88.62:32951)
0928 23:54:35.894289 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:35.894296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:35.895554 (+ 1258us) server_negotiation.cc:407] Connection header received
0928 23:54:35.896918 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:35.896921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:35.897041 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:35.897129 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:35.898672 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.899273 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.900771 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.901019 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.901139 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.904751 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:35.904755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:35.904758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:35.904800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:35.906624 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.907293 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.907295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.907297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.907395 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.907772 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.907775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.907777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.907981 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:35.908095 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:35.908696 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 23:54:35.908833 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:54:35.909823 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.894884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.2:53441)
0928 23:54:35.895365 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:35.895411 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:35.896686 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:35.897436 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:35.897444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:35.897892 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:35.898509 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.898524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.899496 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.899501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.900642 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.900652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.901198 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.901202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.901393 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.902647 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:35.902667 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:35.904506 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:35.906779 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.906785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.906801 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.907144 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.907501 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.907504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.907506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.907659 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.908093 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:35.908100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:35.908447 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 23:54:35.908637 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 23:54:35.908988 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260928 23:54:35.911134 22290 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32951
I20260928 23:54:35.911317 22290 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:35.911841 22290 heartbeater.cc:511] Master 127.20.88.62:32951 requested a full tablet report, sending...
I20260928 23:54:35.912973 21947 ts_manager.cc:194] Registered new tserver with Master: d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063)
I20260928 23:54:35.913723 21947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:53441
I20260928 23:54:35.915349 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22159
I20260928 23:54:35.915472 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:35.915725 20832 external_mini_cluster.cc:1428] Setting key 4e83c532eaa077b4a4841410c03f1f6d
I20260928 23:54:35.921343 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.913485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:43243 (local address 127.20.88.62:32951)
0928 23:54:35.913628 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:35.913633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:35.913836 (+ 203us) server_negotiation.cc:407] Connection header received
0928 23:54:35.913950 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:35.913953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:35.913992 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:35.914084 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:35.914543 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.915069 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.916159 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.916531 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.916642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.918014 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:35.918018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:35.918020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:35.918047 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:35.919558 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.920011 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.920014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.920016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.920100 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:35.920429 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:35.920433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:35.920435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:35.920664 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:35.920779 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:35.921051 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 23:54:35.921169 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:54:35.921379 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:35.913422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.2:43243)
0928 23:54:35.913734 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:35.913752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:35.913863 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:35.914102 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:35.914106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:35.914322 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:35.914423 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.914429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.915242 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.915245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.916027 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:35.916037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:35.916673 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:35.916676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:35.916759 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:35.917474 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:35.917509 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:35.917864 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:35.919689 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.919693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.919696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.919907 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.920196 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:35.920199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:35.920201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:35.920323 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:35.920769 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:35.920772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:35.920883 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 23:54:35.920994 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:54:35.921162 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":3,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:35.947381 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32951
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:36.065439 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:36.065742 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:36.065788 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:36.070134 22299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:36.070220 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:36.070262 22299 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:36.070327 22299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:36.076519 22299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:36.077745 22299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:36.078889 22299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:36.089382 22304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:36.089416 22307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:36.089483 22299 server_base.cc:1034] running on GCE node
W20260928 23:54:36.089397 22305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:36.090184 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:36.090910 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:36.092108 22299 hybrid_clock.cc:648] HybridClock initialized: now 1790639676092086 us; error 36 us; skew 500 ppm
Sep 28 23:54:36 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639676, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:36.095331 22299 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:36.096416 22299 webserver.cc:460] Webserver started at http://127.20.88.3:40691/ using document root <none> and password file <none>
I20260928 23:54:36.097000 22299 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:36.097064 22299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:36.097235 22299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:36.099416 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "fd48162a820543ea91abb5ed3fa55d99"
format_stamp: "Formatted at 2026-09-28 23:54:36 on dist-test-slave-wvpw"
server_key: "038cf0dc1b927a496fb9515d4503f4f6"
server_key_iv: "bbce3e6b0e72d305c6921c98a4a4e995"
server_key_version: "encryptionkey@0"
I20260928 23:54:36.099973 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "fd48162a820543ea91abb5ed3fa55d99"
format_stamp: "Formatted at 2026-09-28 23:54:36 on dist-test-slave-wvpw"
server_key: "038cf0dc1b927a496fb9515d4503f4f6"
server_key_iv: "bbce3e6b0e72d305c6921c98a4a4e995"
server_key_version: "encryptionkey@0"
I20260928 23:54:36.104095 22299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260928 23:54:36.106750 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:36.108119 22299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:36.108297 22299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "fd48162a820543ea91abb5ed3fa55d99"
format_stamp: "Formatted at 2026-09-28 23:54:36 on dist-test-slave-wvpw"
server_key: "038cf0dc1b927a496fb9515d4503f4f6"
server_key_iv: "bbce3e6b0e72d305c6921c98a4a4e995"
server_key_version: "encryptionkey@0"
I20260928 23:54:36.108430 22299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:36.130688 22299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:36.134222 22299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:36.134449 22299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:36.136049 22299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:36.136124 22299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:36.136200 22299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:36.136269 22299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 23:54:36 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639676, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:36.147467 22299 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:42183
I20260928 23:54:36.147562 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:42183 every 8 connection(s)
I20260928 23:54:36.148821 22299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:36.151959 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.136833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53583 (local address 127.20.88.62:32951)
0928 23:54:36.136999 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.137005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.138608 (+ 1603us) server_negotiation.cc:407] Connection header received
0928 23:54:36.139991 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.139994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.140057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.140176 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:36.142192 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.142843 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.144397 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.144846 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.144973 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.148226 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:36.148253 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:36.148257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:36.148292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:36.150029 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.150498 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.150501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.150502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.150583 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.150880 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.150882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.150883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.151068 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:36.151144 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.151652 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.151783 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:54:36.152802 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.137911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.3:53583)
0928 23:54:36.138443 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.138496 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.139738 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.140479 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.140493 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.141104 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:36.141980 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.142010 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.142992 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.142996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.144180 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.144191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.145026 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.145028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.145271 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.146064 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:36.146094 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:36.147957 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:36.150135 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.150139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.150152 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.150417 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.150664 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.150667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.150668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.150794 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.151154 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:36.151158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:36.151399 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.151562 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.151827 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260928 23:54:36.154289 22430 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32951
I20260928 23:54:36.154490 22430 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:36.154997 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22299
I20260928 23:54:36.155115 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:36.155205 22430 heartbeater.cc:511] Master 127.20.88.62:32951 requested a full tablet report, sending...
I20260928 23:54:36.155441 20832 external_mini_cluster.cc:1428] Setting key 29a6daf631b8506345937b776f29dedc
I20260928 23:54:36.156397 21947 ts_manager.cc:194] Registered new tserver with Master: fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3:42183)
I20260928 23:54:36.157325 21947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:53583
I20260928 23:54:36.158802 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:54:36.164676 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.156584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:60693 (local address 127.20.88.62:32951)
0928 23:54:36.156709 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.156714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.156894 (+ 180us) server_negotiation.cc:407] Connection header received
0928 23:54:36.157010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.157013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.157065 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.157132 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:36.157710 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.158443 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.159456 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.159692 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.159816 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.160987 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:36.160991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:36.160994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:36.161019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:36.162689 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.163170 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.163176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.163181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.163279 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.163553 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.163559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.163564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.163883 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:36.163997 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.164388 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.164501 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260928 23:54:36.164909 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.156485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.20.88.3:60693)
0928 23:54:36.156800 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.156815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.156927 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.157168 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.157170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.157410 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:36.157564 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.157572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.158603 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.158606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.159330 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.159341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.159811 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.159815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.159920 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.160492 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:36.160504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:36.160841 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:36.162846 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.162848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.162850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.163057 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.163375 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.163377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.163379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.163456 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.164048 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:36.164054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:36.164165 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.164323 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.164504 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
Sep 28 23:54:36 dist-test-slave-wvpw krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:54:36 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:36.187062 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.178148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.20.88.62:32951)
0928 23:54:36.178291 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.178295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.178309 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:54:36.178354 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.178356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.178397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.178495 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:36.179239 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.179766 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.180722 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.180955 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.181081 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.183617 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:36.183622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:36.183625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:36.183663 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:36.185250 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.185740 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.185744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.185746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.185842 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.186134 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.186139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.186141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.186346 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:36.186517 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.186752 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.186892 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:54:36.190491 21947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:36.194054 21947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:36.210114 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.204522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.20.88.1:42445)
0928 23:54:36.204950 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.204960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.204986 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:54:36.205069 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.205074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.205302 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.205466 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.206002 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.206738 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.208460 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209349 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209473 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.209660 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.209843 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.209971 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":99,"threads_started":1}
I20260928 23:54:36.210114 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.204038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42445 (local address 127.0.0.1:47710)
0928 23:54:36.204845 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.204858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.204935 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.205632 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.205635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.205673 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.205873 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.205879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.206875 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.206879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.208315 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.208327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209557 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.209651 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.209673 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.209790 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.209940 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":65,"spinlock_wait_cycles":9856,"thread_start_us":83,"threads_started":1}
I20260928 23:54:36.210170 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.203620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:42183 (local address 127.0.0.1:51930)
0928 23:54:36.203994 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.204045 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.204177 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.205620 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.205627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.205674 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.205956 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.205966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.207370 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.207374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.208593 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.208605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209742 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.209861 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.209885 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.209960 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.210030 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":183,"spinlock_wait_cycles":28544,"thread_start_us":123,"threads_started":1}
I20260928 23:54:36.210253 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.204418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.20.88.3:42183)
0928 23:54:36.205055 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.205067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.205089 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:54:36.205162 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.205167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.205375 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.205759 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.206074 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.207205 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.208723 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209474 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209563 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.209695 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.210010 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.210103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":116,"threads_started":1}
I20260928 23:54:36.213403 22364 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca2697b8834ae387a8475e98b9d8af (DEFAULT_TABLE table=test-table [id=bcb65088a51e482da18aac43b996badc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.213416 22084 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca2697b8834ae387a8475e98b9d8af (DEFAULT_TABLE table=test-table [id=bcb65088a51e482da18aac43b996badc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.214371 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca2697b8834ae387a8475e98b9d8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.214382 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca2697b8834ae387a8475e98b9d8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.211189 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.203680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:34063 (local address 127.0.0.1:38198)
0928 23:54:36.204564 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.204578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.204670 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.205620 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.205627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.205673 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.205889 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.205897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.207339 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.207344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.208499 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.208506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209423 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.209579 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.210283 (+ 704us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.210936 (+ 653us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.211025 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":721,"spinlock_wait_cycles":22016,"thread_start_us":112,"threads_started":1}
I20260928 23:54:36.221410 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.204498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.20.88.2:34063)
0928 23:54:36.205180 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.205190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.205209 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:54:36.205274 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.205280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.205452 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.205632 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.206031 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.207205 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.208616 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.209277 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.209662 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.209843 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.221002 (+ 11159us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.221229 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":103,"threads_started":1}
I20260928 23:54:36.221675 22454 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Bootstrap starting.
I20260928 23:54:36.221985 22453 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Bootstrap starting.
I20260928 23:54:36.223937 22453 tablet_bootstrap.cc:654] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.223928 22454 tablet_bootstrap.cc:654] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.224766 22453 log.cc:824] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:36.224807 22454 log.cc:824] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:36.224782 22224 tablet_service.cc:1467] Processing CreateTablet for tablet c8ca2697b8834ae387a8475e98b9d8af (DEFAULT_TABLE table=test-table [id=bcb65088a51e482da18aac43b996badc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.225955 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ca2697b8834ae387a8475e98b9d8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.226846 22454 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: No bootstrap required, opened a new log
I20260928 23:54:36.227038 22454 ts_tablet_manager.cc:1397] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 23:54:36.228363 22453 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: No bootstrap required, opened a new log
I20260928 23:54:36.228782 22453 ts_tablet_manager.cc:1397] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 23:54:36.231240 22454 raft_consensus.cc:348] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.231542 22454 raft_consensus.cc:374] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.231604 22454 raft_consensus.cc:729] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd48162a820543ea91abb5ed3fa55d99, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.232084 22454 consensus_queue.cc:260] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.233198 22454 ts_tablet_manager.cc:1428] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:54:36.233222 22453 raft_consensus.cc:348] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.233554 22453 raft_consensus.cc:374] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.233590 22430 heartbeater.cc:503] Master 127.20.88.62:32951 was elected leader, sending a full tablet report...
I20260928 23:54:36.233608 22453 raft_consensus.cc:729] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5004b793d24050a1dc6d45889aa4ac, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.234097 22453 consensus_queue.cc:260] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.235118 22453 ts_tablet_manager.cc:1428] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:54:36.235369 22458 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Bootstrap starting.
I20260928 23:54:36.235471 22150 heartbeater.cc:503] Master 127.20.88.62:32951 was elected leader, sending a full tablet report...
I20260928 23:54:36.237854 22458 tablet_bootstrap.cc:654] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.238749 22458 log.cc:824] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:36.240747 22458 tablet_bootstrap.cc:492] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: No bootstrap required, opened a new log
I20260928 23:54:36.240936 22458 ts_tablet_manager.cc:1397] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:54:36.244179 22458 raft_consensus.cc:348] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.244562 22458 raft_consensus.cc:374] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.244627 22458 raft_consensus.cc:729] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d8fa3b59e24aa5b73a9c53093ae047, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.245095 22458 consensus_queue.cc:260] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.245793 22290 heartbeater.cc:503] Master 127.20.88.62:32951 was elected leader, sending a full tablet report...
I20260928 23:54:36.246243 22458 ts_tablet_manager.cc:1428] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:54:36.391732 22457 raft_consensus.cc:482] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:36.391989 22457 raft_consensus.cc:504] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.393430 22457 leader_election.cc:288] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063), 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445)
I20260928 23:54:36.397589 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.393713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:34063 (local address 127.20.88.3:52145)
0928 23:54:36.393901 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.393918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.394006 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.394431 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.394434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.394452 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.394575 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.394581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.395467 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.395473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.396367 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.396376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.397062 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.397065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.397221 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.397240 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.397344 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.397428 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":22}
I20260928 23:54:36.397589 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.394028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:52145 (local address 127.20.88.2:34063)
0928 23:54:36.394217 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.394221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.394233 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:36.394277 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.394279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.394323 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.394419 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.394698 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.395348 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.396492 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.396931 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.397058 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.397144 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.397371 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.397445 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 23:54:36.398384 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca2697b8834ae387a8475e98b9d8af" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" is_pre_election: true
I20260928 23:54:36.398548 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.393713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42445 (local address 127.20.88.3:53209)
0928 23:54:36.394550 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.394562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.394697 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.395002 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.395004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.395020 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.395560 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.395568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.396354 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.396360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.397203 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.397212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.398111 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.398115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.398210 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.398233 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.398320 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.398392 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":485,"thread_start_us":160,"threads_started":1}
I20260928 23:54:36.398520 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.393992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53209 (local address 127.20.88.1:42445)
0928 23:54:36.394123 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.394129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.394653 (+ 524us) server_negotiation.cc:407] Connection header received
0928 23:54:36.394815 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.394818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.394873 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.394989 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.395686 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.396184 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.397332 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.397952 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.398058 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.398115 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.398341 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.398420 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:54:36.398792 22244 raft_consensus.cc:2454] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 0.
I20260928 23:54:36.399298 22317 leader_election.cc:302] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8d8fa3b59e24aa5b73a9c53093ae047, fd48162a820543ea91abb5ed3fa55d99; no voters:
I20260928 23:54:36.399302 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca2697b8834ae387a8475e98b9d8af" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac" is_pre_election: true
I20260928 23:54:36.399614 22104 raft_consensus.cc:2454] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 0.
I20260928 23:54:36.399618 22457 raft_consensus.cc:2781] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:36.399711 22457 raft_consensus.cc:482] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:36.399775 22457 raft_consensus.cc:3037] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:54:36.400815 22431 tablet.cc:2369] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:36.401384 22457 raft_consensus.cc:504] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
I20260928 23:54:36.401832 22457 leader_election.cc:288] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 election: Requested vote from peers d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063), 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445)
I20260928 23:54:36.402302 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca2697b8834ae387a8475e98b9d8af" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047"
I20260928 23:54:36.402380 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ca2697b8834ae387a8475e98b9d8af" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac"
I20260928 23:54:36.402469 22244 raft_consensus.cc:3037] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.402503 22104 raft_consensus.cc:3037] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.403909 22244 raft_consensus.cc:2454] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 1.
I20260928 23:54:36.403909 22104 raft_consensus.cc:2454] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 1.
I20260928 23:54:36.404320 22317 leader_election.cc:302] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8d8fa3b59e24aa5b73a9c53093ae047, fd48162a820543ea91abb5ed3fa55d99; no voters:
I20260928 23:54:36.404567 22457 raft_consensus.cc:2781] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:36.404889 22457 raft_consensus.cc:686] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 1 LEADER]: Becoming Leader. State: Replica: fd48162a820543ea91abb5ed3fa55d99, State: Running, Role: LEADER
I20260928 23:54:36.405261 22457 consensus_queue.cc:237] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } }
W20260928 23:54:36.407222 22291 tablet.cc:2369] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:36.408891 21943 catalog_manager.cc:5495] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3). New cstate: current_term: 1 leader_uuid: "fd48162a820543ea91abb5ed3fa55d99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } health_report { overall_health: UNKNOWN } } }
W20260928 23:54:36.422423 22151 tablet.cc:2369] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:36.454303 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.450461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.20.88.3:42183)
0928 23:54:36.450612 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.450615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.450631 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:54:36.450859 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.450862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.450923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.451013 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:36.451313 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.451839 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.452779 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.453018 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.453110 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.453441 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:36.453911 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:36.453999 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.454119 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.454180 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:54:36.460711 22104 raft_consensus.cc:1261] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Refusing update from remote peer fd48162a820543ea91abb5ed3fa55d99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:36.460711 22244 raft_consensus.cc:1261] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Refusing update from remote peer fd48162a820543ea91abb5ed3fa55d99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:36.461513 22463 consensus_queue.cc:1035] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:36.461819 22457 consensus_queue.cc:1035] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:36.472064 22467 mvcc.cc:205] Tried to move back new op lower bound from 7334460114775113728 to 7334460114558976000. Current Snapshot: MvccSnapshot[applied={T|T < 7334460114775113728 or (T in {7334460114775113728})}]
I20260928 23:54:36.472360 22468 mvcc.cc:205] Tried to move back new op lower bound from 7334460114775113728 to 7334460114558976000. Current Snapshot: MvccSnapshot[applied={T|T < 7334460114775113728}]
I20260928 23:54:36.483186 21943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:36.483419 21943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:36.486306 21943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Sending DeleteTablet for 3 replicas of tablet c8ca2697b8834ae387a8475e98b9d8af
I20260928 23:54:36.487264 22224 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ca2697b8834ae387a8475e98b9d8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260928 23:54:36.487272 22364 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ca2697b8834ae387a8475e98b9d8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260928 23:54:36.487819 22476 tablet_replica.cc:330] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: stopping tablet replica
I20260928 23:54:36.487824 22084 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ca2697b8834ae387a8475e98b9d8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:47710
I20260928 23:54:36.488199 22476 raft_consensus.cc:2229] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:36.488421 22478 tablet_replica.cc:330] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: stopping tablet replica
I20260928 23:54:36.488588 22477 tablet_replica.cc:330] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: stopping tablet replica
I20260928 23:54:36.488849 22477 raft_consensus.cc:2229] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:36.488850 22478 raft_consensus.cc:2229] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:36.489254 22477 raft_consensus.cc:2258] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:36.489271 22478 raft_consensus.cc:2258] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:36.489742 21943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:54:36.490073 22476 raft_consensus.cc:2258] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:54:36.490264 21943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:36.491277 22478 ts_tablet_manager.cc:1905] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:36.491746 22476 ts_tablet_manager.cc:1905] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:36.494378 22478 ts_tablet_manager.cc:1918] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:36.494499 22478 log.cc:1196] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/c8ca2697b8834ae387a8475e98b9d8af
I20260928 23:54:36.494791 22476 ts_tablet_manager.cc:1918] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:36.494884 22478 ts_tablet_manager.cc:1939] T c8ca2697b8834ae387a8475e98b9d8af P 7a5004b793d24050a1dc6d45889aa4ac: Deleting consensus metadata
I20260928 23:54:36.495007 22476 log.cc:1196] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/c8ca2697b8834ae387a8475e98b9d8af
I20260928 23:54:36.495373 22476 ts_tablet_manager.cc:1939] T c8ca2697b8834ae387a8475e98b9d8af P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting consensus metadata
I20260928 23:54:36.496472 21934 catalog_manager.cc:4841] TS 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445): tablet c8ca2697b8834ae387a8475e98b9d8af (table test-table [id=bcb65088a51e482da18aac43b996badc]) successfully deleted
I20260928 23:54:36.496636 21933 catalog_manager.cc:4841] TS d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063): tablet c8ca2697b8834ae387a8475e98b9d8af (table test-table [id=bcb65088a51e482da18aac43b996badc]) successfully deleted
I20260928 23:54:36.497275 22084 tablet_service.cc:1467] Processing CreateTablet for tablet c85e0755366542b8b7a5af98630453fd (DEFAULT_TABLE table=test-table [id=019533ab294c48b48a961d704d7b6804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.497599 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85e0755366542b8b7a5af98630453fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.497538 22364 tablet_service.cc:1467] Processing CreateTablet for tablet c85e0755366542b8b7a5af98630453fd (DEFAULT_TABLE table=test-table [id=019533ab294c48b48a961d704d7b6804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.497574 22224 tablet_service.cc:1467] Processing CreateTablet for tablet c85e0755366542b8b7a5af98630453fd (DEFAULT_TABLE table=test-table [id=019533ab294c48b48a961d704d7b6804]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:36.497802 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85e0755366542b8b7a5af98630453fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.497808 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85e0755366542b8b7a5af98630453fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.500080 22453 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Bootstrap starting.
I20260928 23:54:36.500151 22458 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Bootstrap starting.
I20260928 23:54:36.500366 22454 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Bootstrap starting.
I20260928 23:54:36.501202 22458 tablet_bootstrap.cc:654] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.501204 22453 tablet_bootstrap.cc:654] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.501266 22454 tablet_bootstrap.cc:654] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.503065 22454 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: No bootstrap required, opened a new log
I20260928 23:54:36.503074 22458 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: No bootstrap required, opened a new log
I20260928 23:54:36.503153 22454 ts_tablet_manager.cc:1397] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 23:54:36.503158 22458 ts_tablet_manager.cc:1397] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:36.503587 22477 ts_tablet_manager.cc:1905] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:36.503615 22454 raft_consensus.cc:348] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.503623 22458 raft_consensus.cc:348] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.503727 22454 raft_consensus.cc:374] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.503729 22458 raft_consensus.cc:374] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.503759 22458 raft_consensus.cc:729] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d8fa3b59e24aa5b73a9c53093ae047, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.503758 22454 raft_consensus.cc:729] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd48162a820543ea91abb5ed3fa55d99, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.503881 22454 consensus_queue.cc:260] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.503893 22458 consensus_queue.cc:260] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.504128 22454 ts_tablet_manager.cc:1428] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:36.504158 22458 ts_tablet_manager.cc:1428] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:36.506573 22453 tablet_bootstrap.cc:492] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: No bootstrap required, opened a new log
I20260928 23:54:36.506682 22453 ts_tablet_manager.cc:1397] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 23:54:36.507133 22477 ts_tablet_manager.cc:1918] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:36.507241 22477 log.cc:1196] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/c8ca2697b8834ae387a8475e98b9d8af
I20260928 23:54:36.507283 22453 raft_consensus.cc:348] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.507417 22453 raft_consensus.cc:374] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.507462 22453 raft_consensus.cc:729] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5004b793d24050a1dc6d45889aa4ac, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.507606 22477 ts_tablet_manager.cc:1939] T c8ca2697b8834ae387a8475e98b9d8af P fd48162a820543ea91abb5ed3fa55d99: Deleting consensus metadata
I20260928 23:54:36.507642 22453 consensus_queue.cc:260] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.507951 22453 ts_tablet_manager.cc:1428] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 23:54:36.508970 21931 catalog_manager.cc:4841] TS fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3:42183): tablet c8ca2697b8834ae387a8475e98b9d8af (table test-table [id=bcb65088a51e482da18aac43b996badc]) successfully deleted
I20260928 23:54:36.713069 22457 raft_consensus.cc:482] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:36.713222 22457 raft_consensus.cc:504] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.713729 22457 leader_election.cc:288] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445), d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063)
I20260928 23:54:36.714326 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85e0755366542b8b7a5af98630453fd" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac" is_pre_election: true
I20260928 23:54:36.714318 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85e0755366542b8b7a5af98630453fd" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" is_pre_election: true
I20260928 23:54:36.714504 22244 raft_consensus.cc:2454] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 0.
I20260928 23:54:36.714504 22104 raft_consensus.cc:2454] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 0.
I20260928 23:54:36.714923 22318 leader_election.cc:302] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5004b793d24050a1dc6d45889aa4ac, fd48162a820543ea91abb5ed3fa55d99; no voters:
I20260928 23:54:36.715150 22457 raft_consensus.cc:2781] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:36.715214 22457 raft_consensus.cc:482] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:36.715274 22457 raft_consensus.cc:3037] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.716418 22457 raft_consensus.cc:504] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.716796 22457 leader_election.cc:288] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 election: Requested vote from peers 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445), d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063)
I20260928 23:54:36.717252 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85e0755366542b8b7a5af98630453fd" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047"
I20260928 23:54:36.717252 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85e0755366542b8b7a5af98630453fd" candidate_uuid: "fd48162a820543ea91abb5ed3fa55d99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac"
I20260928 23:54:36.717396 22104 raft_consensus.cc:3037] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.717396 22244 raft_consensus.cc:3037] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.718595 22244 raft_consensus.cc:2454] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 1.
I20260928 23:54:36.718593 22104 raft_consensus.cc:2454] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd48162a820543ea91abb5ed3fa55d99 in term 1.
I20260928 23:54:36.718952 22318 leader_election.cc:302] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a5004b793d24050a1dc6d45889aa4ac, d8d8fa3b59e24aa5b73a9c53093ae047, fd48162a820543ea91abb5ed3fa55d99; no voters:
I20260928 23:54:36.719125 22457 raft_consensus.cc:2781] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:36.719215 22457 raft_consensus.cc:686] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 1 LEADER]: Becoming Leader. State: Replica: fd48162a820543ea91abb5ed3fa55d99, State: Running, Role: LEADER
I20260928 23:54:36.719384 22457 consensus_queue.cc:237] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.721329 21947 catalog_manager.cc:5495] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3). New cstate: current_term: 1 leader_uuid: "fd48162a820543ea91abb5ed3fa55d99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } health_report { overall_health: HEALTHY } } }
Sep 28 23:54:36 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639675, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:36.784184 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.774271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32951 (local address 127.0.0.1:47670)
0928 23:54:36.774592 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.774613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.774728 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.775018 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.775020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.775500 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:36.775608 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.775615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.776757 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.776760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.777420 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.777430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.777880 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.777884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.777949 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.778600 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:36.778612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:36.780314 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:36.782599 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.782609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.782616 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.782885 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.783163 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:36.783165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:36.783167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:36.783245 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:36.783683 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:36.783688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:36.783800 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.783893 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.784037 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
I20260928 23:54:36.784294 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.774327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.20.88.62:32951)
0928 23:54:36.774595 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.774601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.774695 (+ 94us) server_negotiation.cc:407] Connection header received
0928 23:54:36.774843 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.774846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.774888 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.774979 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:36.775799 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.776648 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.777547 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.777768 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.777857 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.780543 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:36.780549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:36.780554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:36.780587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:36.782483 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.782988 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.782991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.782992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.783075 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:36.783360 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:36.783363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:36.783364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:36.783579 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:36.783673 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.783996 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.784113 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260928 23:54:36.788746 21947 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:47670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:54:36.789639 21947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:36.795303 22084 tablet_service.cc:1467] Processing CreateTablet for tablet 80ee4195e9a74cb9ad601034b2d1f13b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdad0639a5394f1c8bd74aa0103ecc31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:36.795389 22224 tablet_service.cc:1467] Processing CreateTablet for tablet 80ee4195e9a74cb9ad601034b2d1f13b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdad0639a5394f1c8bd74aa0103ecc31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:36.795814 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ee4195e9a74cb9ad601034b2d1f13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.795828 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ee4195e9a74cb9ad601034b2d1f13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.798475 22453 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: Bootstrap starting.
I20260928 23:54:36.798825 22458 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: Bootstrap starting.
I20260928 23:54:36.799151 22364 tablet_service.cc:1467] Processing CreateTablet for tablet 80ee4195e9a74cb9ad601034b2d1f13b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdad0639a5394f1c8bd74aa0103ecc31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:36.799474 22453 tablet_bootstrap.cc:654] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.799551 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ee4195e9a74cb9ad601034b2d1f13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:36.799793 22458 tablet_bootstrap.cc:654] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.801100 22458 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: No bootstrap required, opened a new log
I20260928 23:54:36.801220 22458 ts_tablet_manager.cc:1397] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:54:36.801267 22453 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: No bootstrap required, opened a new log
I20260928 23:54:36.801352 22453 ts_tablet_manager.cc:1397] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:36.801753 22458 raft_consensus.cc:348] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.801779 22453 raft_consensus.cc:348] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.801887 22458 raft_consensus.cc:374] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.801899 22453 raft_consensus.cc:374] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.801921 22458 raft_consensus.cc:729] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d8fa3b59e24aa5b73a9c53093ae047, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.801954 22453 raft_consensus.cc:729] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5004b793d24050a1dc6d45889aa4ac, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.802073 22458 consensus_queue.cc:260] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.802095 22453 consensus_queue.cc:260] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.802354 22458 ts_tablet_manager.cc:1428] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 23:54:36.802363 22459 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:36.802501 22459 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:36.802603 22461 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:36.802753 22461 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:36.803077 22454 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: Bootstrap starting.
I20260928 23:54:36.804082 22454 tablet_bootstrap.cc:654] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:36.804811 22453 ts_tablet_manager.cc:1428] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:36.805701 22454 tablet_bootstrap.cc:492] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: No bootstrap required, opened a new log
I20260928 23:54:36.805814 22454 ts_tablet_manager.cc:1397] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:36.806260 22454 raft_consensus.cc:348] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.806386 22454 raft_consensus.cc:374] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:36.806427 22454 raft_consensus.cc:729] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd48162a820543ea91abb5ed3fa55d99, State: Initialized, Role: FOLLOWER
I20260928 23:54:36.806525 22454 consensus_queue.cc:260] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.806787 22454 ts_tablet_manager.cc:1428] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:36.806743 22457 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:36.806869 22457 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:36.966323 22461 raft_consensus.cc:482] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:36.966478 22461 raft_consensus.cc:504] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.967685 22461 leader_election.cc:288] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445), fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3:42183)
I20260928 23:54:36.972316 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.967953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42445 (local address 127.20.88.2:34251)
0928 23:54:36.968421 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.968443 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.968663 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.969012 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.969014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.969029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.969157 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.969165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.969912 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.969918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.970742 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.970751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.971750 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.971754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.971900 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.971929 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.972013 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.972089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":50,"thread_start_us":119,"threads_started":1}
I20260928 23:54:36.972379 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.968114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:34251 (local address 127.20.88.1:42445)
0928 23:54:36.968348 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.968354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.968549 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:54:36.968808 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.968812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.968888 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.968988 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.969270 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.969785 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.970866 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.971475 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.971576 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.971639 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.972070 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.972163 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":54,"threads_started":1}
I20260928 23:54:36.972689 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.967950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:42183 (local address 127.20.88.2:50809)
0928 23:54:36.968333 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.968358 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.968479 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.968892 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.968895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.968918 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:36.969062 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.969070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.969932 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.969937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.970927 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.970937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.972081 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.972084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.972168 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.972191 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:54:36.972317 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:54:36.972458 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260928 23:54:36.973038 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.968067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50809 (local address 127.20.88.3:42183)
0928 23:54:36.968368 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.968373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.968418 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:54:36.968686 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.968692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.968757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.968842 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:36.969180 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.969754 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.971137 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.971910 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.972029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.972089 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:36.972344 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:54:36.972912 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260928 23:54:36.972976 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ee4195e9a74cb9ad601034b2d1f13b" candidate_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac" is_pre_election: true
I20260928 23:54:36.973156 22104 raft_consensus.cc:2454] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d8fa3b59e24aa5b73a9c53093ae047 in term 0.
I20260928 23:54:36.973560 22178 leader_election.cc:302] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5004b793d24050a1dc6d45889aa4ac, d8d8fa3b59e24aa5b73a9c53093ae047; no voters:
I20260928 23:54:36.973573 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ee4195e9a74cb9ad601034b2d1f13b" candidate_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd48162a820543ea91abb5ed3fa55d99" is_pre_election: true
I20260928 23:54:36.973822 22384 raft_consensus.cc:2454] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d8fa3b59e24aa5b73a9c53093ae047 in term 0.
I20260928 23:54:36.973865 22461 raft_consensus.cc:2781] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:36.973922 22461 raft_consensus.cc:482] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:36.973987 22461 raft_consensus.cc:3037] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.975114 22461 raft_consensus.cc:504] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.975651 22461 leader_election.cc:288] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [CANDIDATE]: Term 1 election: Requested vote from peers 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445), fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3:42183)
I20260928 23:54:36.976150 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ee4195e9a74cb9ad601034b2d1f13b" candidate_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd48162a820543ea91abb5ed3fa55d99"
I20260928 23:54:36.976169 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ee4195e9a74cb9ad601034b2d1f13b" candidate_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a5004b793d24050a1dc6d45889aa4ac"
I20260928 23:54:36.976346 22384 raft_consensus.cc:3037] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.976378 22104 raft_consensus.cc:3037] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:36.977455 22104 raft_consensus.cc:2454] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d8fa3b59e24aa5b73a9c53093ae047 in term 1.
I20260928 23:54:36.977562 22384 raft_consensus.cc:2454] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d8fa3b59e24aa5b73a9c53093ae047 in term 1.
I20260928 23:54:36.977859 22178 leader_election.cc:302] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a5004b793d24050a1dc6d45889aa4ac, d8d8fa3b59e24aa5b73a9c53093ae047; no voters:
I20260928 23:54:36.978049 22461 raft_consensus.cc:2781] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:36.978256 22461 raft_consensus.cc:686] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 LEADER]: Becoming Leader. State: Replica: d8d8fa3b59e24aa5b73a9c53093ae047, State: Running, Role: LEADER
I20260928 23:54:36.978535 22461 consensus_queue.cc:237] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } }
I20260928 23:54:36.979916 22497 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: TxnStatusTablet state changed. Reason: New leader d8d8fa3b59e24aa5b73a9c53093ae047. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:36.980051 22497 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:36.980377 22499 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:54:36.980557 22499 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:54:36.981472 21947 catalog_manager.cc:5495] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20260928 23:54:37.001008 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.996452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:34063 (local address 127.0.0.1:38210)
0928 23:54:36.996608 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:36.996627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:36.996760 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:36.997176 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:36.997179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:36.997190 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:36.997356 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.997364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.998445 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.998447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.999077 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:36.999088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.999680 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.999683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:36.999752 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.999807 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.000703 (+ 896us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.000719 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 23:54:37.000792 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:54:37.000879 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 23:54:37.001055 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:36.996548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.20.88.2:34063)
0928 23:54:36.996835 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:36.996843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:36.996860 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:54:36.996942 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:36.996945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:36.997010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:36.997133 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:36.997475 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.998314 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.999194 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:36.999564 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:36.999667 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:36.999950 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.000579 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.000692 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:37.000823 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:54:37.000887 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260928 23:54:37.004453 22104 raft_consensus.cc:1261] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Refusing update from remote peer d8d8fa3b59e24aa5b73a9c53093ae047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:37.004679 22384 raft_consensus.cc:1261] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Refusing update from remote peer d8d8fa3b59e24aa5b73a9c53093ae047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:37.004985 22497 consensus_queue.cc:1035] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:37.005360 22461 consensus_queue.cc:1035] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:37.006760 22459 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: TxnStatusTablet state changed. Reason: New leader d8d8fa3b59e24aa5b73a9c53093ae047. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.006881 22459 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:37.008482 22457 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: TxnStatusTablet state changed. Reason: New leader d8d8fa3b59e24aa5b73a9c53093ae047. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.008610 22457 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:37.009930 22461 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.010051 22461 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:37.010285 22459 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.010385 22459 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P 7a5004b793d24050a1dc6d45889aa4ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:37.012066 22497 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.012192 22497 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P d8d8fa3b59e24aa5b73a9c53093ae047: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:37.012871 22457 tablet_replica.cc:439] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 } } peers { permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 } } peers { permanent_uuid: "fd48162a820543ea91abb5ed3fa55d99" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 42183 } } }
I20260928 23:54:37.012987 22457 tablet_replica.cc:442] T 80ee4195e9a74cb9ad601034b2d1f13b P fd48162a820543ea91abb5ed3fa55d99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:37.014667 22506 mvcc.cc:205] Tried to move back new op lower bound from 7334460117004931072 to 7334460116906889216. Current Snapshot: MvccSnapshot[applied={T|T < 7334460117004931072}]
I20260928 23:54:37.028824 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.024986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:34063 (local address 127.20.88.3:41579)
0928 23:54:37.025305 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:37.025318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:37.025436 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:37.025724 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:37.025727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:37.025735 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:37.025834 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:37.025838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.026854 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.026856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:37.027498 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:37.027507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.028087 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.028090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:37.028150 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.028197 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.028565 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.028571 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:54:37.028621 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:54:37.028694 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260928 23:54:37.028832 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.025104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:41579 (local address 127.20.88.2:34063)
0928 23:54:37.025242 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:37.025246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:37.025418 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:54:37.025535 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:37.025538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:37.025592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:37.025681 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:37.025958 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.026736 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.027628 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.027974 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.028081 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.028388 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.028480 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.028555 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:37.028638 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 23:54:37.028695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:54:37.036043 22104 raft_consensus.cc:1261] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Refusing update from remote peer fd48162a820543ea91abb5ed3fa55d99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:37.036074 22244 raft_consensus.cc:1261] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Refusing update from remote peer fd48162a820543ea91abb5ed3fa55d99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:37.036494 22457 consensus_queue.cc:1035] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5004b793d24050a1dc6d45889aa4ac" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:37.036665 22463 consensus_queue.cc:1035] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d8fa3b59e24aa5b73a9c53093ae047" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:37.062013 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.057709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:42183 (local address 127.20.88.2:58375)
0928 23:54:37.058214 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:37.058229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:37.058355 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:37.058633 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:37.058636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:37.058645 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:37.058783 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:37.058790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.059823 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.059826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:37.060698 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:37.060709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.061307 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.061310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:37.061386 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.061439 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.061752 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.061757 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:54:37.061807 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:54:37.061877 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":166,"threads_started":1}
I20260928 23:54:37.062019 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.057905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:58375 (local address 127.20.88.3:42183)
0928 23:54:37.058028 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:37.058033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:37.058313 (+ 280us) server_negotiation.cc:407] Connection header received
0928 23:54:37.058458 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:37.058460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:37.058523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:37.058609 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:37.058907 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.059703 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.060824 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.061170 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.061299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.061578 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:37.061667 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:37.061743 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:37.061819 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 23:54:37.061880 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:54:37.085443 21947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:37.085642 21947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:37.088074 21947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87849a41a94b4b8abec84565f2e5572e: Sending DeleteTablet for 3 replicas of tablet c85e0755366542b8b7a5af98630453fd
I20260928 23:54:37.088786 22084 tablet_service.cc:1514] Processing DeleteTablet for tablet c85e0755366542b8b7a5af98630453fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:37 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:47710
I20260928 23:54:37.088811 22224 tablet_service.cc:1514] Processing DeleteTablet for tablet c85e0755366542b8b7a5af98630453fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:37 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:38198
I20260928 23:54:37.088925 22364 tablet_service.cc:1514] Processing DeleteTablet for tablet c85e0755366542b8b7a5af98630453fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:37 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260928 23:54:37.089275 22523 tablet_replica.cc:330] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: stopping tablet replica
I20260928 23:54:37.089380 22525 tablet_replica.cc:330] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: stopping tablet replica
I20260928 23:54:37.089514 22525 raft_consensus.cc:2229] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:37.089529 22523 raft_consensus.cc:2229] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:37.089716 22525 raft_consensus.cc:2258] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:37.089814 22523 raft_consensus.cc:2258] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:37.090628 22523 ts_tablet_manager.cc:1905] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:37.091065 22525 ts_tablet_manager.cc:1905] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:37.094095 22523 ts_tablet_manager.cc:1918] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:37.094197 22523 log.cc:1196] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/c85e0755366542b8b7a5af98630453fd
I20260928 23:54:37.094187 22525 ts_tablet_manager.cc:1918] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:37.094255 22525 log.cc:1196] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/c85e0755366542b8b7a5af98630453fd
I20260928 23:54:37.094585 22525 ts_tablet_manager.cc:1939] T c85e0755366542b8b7a5af98630453fd P fd48162a820543ea91abb5ed3fa55d99: Deleting consensus metadata
I20260928 23:54:37.094666 22524 tablet_replica.cc:330] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: stopping tablet replica
I20260928 23:54:37.094816 22524 raft_consensus.cc:2229] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:37.094962 22524 raft_consensus.cc:2258] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:37.095486 21931 catalog_manager.cc:4841] TS fd48162a820543ea91abb5ed3fa55d99 (127.20.88.3:42183): tablet c85e0755366542b8b7a5af98630453fd (table test-table [id=019533ab294c48b48a961d704d7b6804]) successfully deleted
I20260928 23:54:37.095840 22524 ts_tablet_manager.cc:1905] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:37.095899 22523 ts_tablet_manager.cc:1939] T c85e0755366542b8b7a5af98630453fd P 7a5004b793d24050a1dc6d45889aa4ac: Deleting consensus metadata
I20260928 23:54:37.096634 21934 catalog_manager.cc:4841] TS 7a5004b793d24050a1dc6d45889aa4ac (127.20.88.1:42445): tablet c85e0755366542b8b7a5af98630453fd (table test-table [id=019533ab294c48b48a961d704d7b6804]) successfully deleted
Sep 28 23:54:37 dist-test-slave-wvpw krb5kdc[21902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639676, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:37.098991 22524 ts_tablet_manager.cc:1918] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:37.099071 22524 log.cc:1196] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/c85e0755366542b8b7a5af98630453fd
I20260928 23:54:37.099368 22524 ts_tablet_manager.cc:1939] T c85e0755366542b8b7a5af98630453fd P d8d8fa3b59e24aa5b73a9c53093ae047: Deleting consensus metadata
I20260928 23:54:37.100169 21933 catalog_manager.cc:4841] TS d8d8fa3b59e24aa5b73a9c53093ae047 (127.20.88.2:34063): tablet c85e0755366542b8b7a5af98630453fd (table test-table [id=019533ab294c48b48a961d704d7b6804]) successfully deleted
I20260928 23:54:37.103808 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.090805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.20.88.1:42445)
0928 23:54:37.090944 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:37.090948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:37.091557 (+ 609us) server_negotiation.cc:407] Connection header received
0928 23:54:37.091741 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:37.091745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:37.091796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:37.091849 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:37.092877 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.093770 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.095605 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.095942 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.096024 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.099726 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:37.099742 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:37.099758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:37.099793 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:37.101881 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:37.102383 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:37.102391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:37.102393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:37.102498 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:37.102788 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:37.102791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:37.102793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:37.103120 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:37.103339 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:37.103497 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:54:37.103632 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 23:54:37.104300 22144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726
I20260928 23:54:37.115309 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:37.106072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.20.88.62:32951)
0928 23:54:37.106191 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:37.106195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:37.106255 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:54:37.106412 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:37.106414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:37.106461 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:37.106528 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:37.107378 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.107927 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.109095 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:37.109356 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:37.109444 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:37.111225 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:37.111228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:37.111230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:37.111266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:37.113286 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:37.113752 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:37.113757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:37.113760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:37.113868 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:37.114204 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:37.114209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:37.114213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:37.114501 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:37.114642 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:37.114949 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:54:37.115100 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:54:37.115743 21947 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678
I20260928 23:54:37.117084 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22019
I20260928 23:54:37.124125 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22159
I20260928 23:54:37.131865 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22299
I20260928 23:54:37.138739 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 21915
2026-09-28T23:54:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3172 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:37 dist-test-slave-wvpw krb5kdc[22532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:37 dist-test-slave-wvpw krb5kdc[22532](info): set up 2 sockets
Sep 28 23:54:37 dist-test-slave-wvpw krb5kdc[22532](info): commencing operation
krb5kdc: starting...
W20260928 23:54:38.192660 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:38 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:38.320732 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35239
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:38.440665 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:38.441056 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:38.441133 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:38.445568 22545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:38.445688 22545 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:54:38.445717 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:38.445763 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:38.445820 22545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:38.445880 22545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:38.451691 22545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:38.453056 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:38.454202 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:38.463467 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:38.463467 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:38.463596 22545 server_base.cc:1034] running on GCE node
W20260928 23:54:38.463478 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:38.464435 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:38.465700 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:38.466920 22545 hybrid_clock.cc:648] HybridClock initialized: now 1790639678466889 us; error 48 us; skew 500 ppm
Sep 28 23:54:38 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:38.470216 22545 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:38.471273 22545 webserver.cc:460] Webserver started at http://127.20.88.62:33689/ using document root <none> and password file <none>
I20260928 23:54:38.471864 22545 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:38.471927 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:38.472112 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:38.474399 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "9b22c77ed83840aca8dea7d99a5d3e4b"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "9240335c24eae15cb3185e951c94b00a"
server_key_iv: "f4f009ed5787624eea7166506164b48f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.474973 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "9b22c77ed83840aca8dea7d99a5d3e4b"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "9240335c24eae15cb3185e951c94b00a"
server_key_iv: "f4f009ed5787624eea7166506164b48f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.479214 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:38.482084 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:38.483705 22545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:38.483846 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "9b22c77ed83840aca8dea7d99a5d3e4b"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "9240335c24eae15cb3185e951c94b00a"
server_key_iv: "f4f009ed5787624eea7166506164b48f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.483968 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:38.501600 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:38.505165 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:38.505340 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:38.513923 22545 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35239
I20260928 23:54:38.514002 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35239 every 8 connection(s)
I20260928 23:54:38.514998 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:38.518199 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:38.518355 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22545
I20260928 23:54:38.518461 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:38.518760 20832 external_mini_cluster.cc:1428] Setting key b86a19760ec0cb76993274bf36be9a20
I20260928 23:54:38.525888 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Bootstrap starting.
Sep 28 23:54:38 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:38.529069 22623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:38.530023 22623 log.cc:824] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:38.532400 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: No bootstrap required, opened a new log
I20260928 23:54:38.536132 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:38.520376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.20.88.62:35239)
0928 23:54:38.520852 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:38.520869 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:38.520906 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:54:38.521726 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:38.521755 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:38.522120 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:38.522448 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:38.523386 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.524556 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.525747 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.526402 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.526535 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:38.529704 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:38.529711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:38.529727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:38.529763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:38.532489 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.533147 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.533152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.533159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.533291 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.533633 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.533635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.533637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.534097 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:38.534254 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:38.534848 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 23:54:38.535249 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260928 23:54:38.537020 22623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } }
I20260928 23:54:38.537317 22623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:38.537371 22623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b22c77ed83840aca8dea7d99a5d3e4b, State: Initialized, Role: FOLLOWER
I20260928 23:54:38.537918 22623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } }
I20260928 23:54:38.538087 22623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:38.538142 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:38.538240 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:38.539354 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } }
I20260928 23:54:38.539688 22623 leader_election.cc:302] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b22c77ed83840aca8dea7d99a5d3e4b; no voters:
I20260928 23:54:38.539996 22623 leader_election.cc:288] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:38.540170 22628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:38.540424 22628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [term 1 LEADER]: Becoming Leader. State: Replica: 9b22c77ed83840aca8dea7d99a5d3e4b, State: Running, Role: LEADER
I20260928 23:54:38.540786 22628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } }
I20260928 23:54:38.541541 22623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:38.542156 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b22c77ed83840aca8dea7d99a5d3e4b. Latest consensus state: current_term: 1 leader_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } } }
I20260928 23:54:38.542296 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:38.542608 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b22c77ed83840aca8dea7d99a5d3e4b" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35239 } } }
I20260928 23:54:38.542712 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:38.544170 22633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:38.547761 22633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:38.554165 22633 catalog_manager.cc:1347] Generated new cluster ID: 7c06df6f7ecb4368b999e058563b7d61
I20260928 23:54:38.554250 22633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:38.593559 22633 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:38.594622 22633 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:38.604883 22633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Generated new TSK 0
I20260928 23:54:38.605708 22633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:38.648061 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35239
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:38.778980 22649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:38.779306 22649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:38.779359 22649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:38.783644 22649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:38.783730 22649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:38.783774 22649 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:54:38.783788 22649 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:38.783823 22649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:38.789788 22649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:38.790952 22649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:38.791983 22649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:38.803277 22654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:38.803460 22657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:38.803443 22649 server_base.cc:1034] running on GCE node
W20260928 23:54:38.803648 22655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:38.804132 22649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:38.804891 22649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:38.806099 22649 hybrid_clock.cc:648] HybridClock initialized: now 1790639678806079 us; error 34 us; skew 500 ppm
Sep 28 23:54:38 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:38.809283 22649 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:38.810271 22649 webserver.cc:460] Webserver started at http://127.20.88.1:36131/ using document root <none> and password file <none>
I20260928 23:54:38.810945 22649 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:38.811048 22649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:38.811308 22649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:38.813601 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "155ffdae490948b9831d9fea14bf18f8"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "1d9c5cf26b940dcbbfd69d3299b55e3a"
server_key_iv: "75053aea91d9cff097d0621dcb90b66f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.814208 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "155ffdae490948b9831d9fea14bf18f8"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "1d9c5cf26b940dcbbfd69d3299b55e3a"
server_key_iv: "75053aea91d9cff097d0621dcb90b66f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.818511 22649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:38.821460 22664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:38.823017 22649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:38.823148 22649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "155ffdae490948b9831d9fea14bf18f8"
format_stamp: "Formatted at 2026-09-28 23:54:38 on dist-test-slave-wvpw"
server_key: "1d9c5cf26b940dcbbfd69d3299b55e3a"
server_key_iv: "75053aea91d9cff097d0621dcb90b66f"
server_key_version: "encryptionkey@0"
I20260928 23:54:38.823377 22649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:38.847839 22649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:38.851665 22649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:38.851900 22649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:38.853567 22649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:38.853626 22649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:38.853670 22649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:38.853732 22649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:38.864934 22649 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:46591
I20260928 23:54:38.865022 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:46591 every 8 connection(s)
Sep 28 23:54:38 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:38.866009 22649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:38.866330 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22649
I20260928 23:54:38.866444 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:38.866775 20832 external_mini_cluster.cc:1428] Setting key 37b676d841be27e195fcb718b39f7410
I20260928 23:54:38.871469 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:38.854769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:37903 (local address 127.20.88.62:35239)
0928 23:54:38.854940 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:38.854945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:38.855985 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 23:54:38.857520 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:38.857524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:38.857587 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:38.857687 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:38.859966 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.860936 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.862889 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.863343 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.863479 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:38.866839 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:38.866843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:38.866847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:38.866886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:38.868731 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.869414 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.869417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.869418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.869496 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.869963 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.869965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.869967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.870333 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:38.870472 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:38.871139 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 23:54:38.871289 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:54:38.872519 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:38.855253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.1:37903)
0928 23:54:38.855778 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:38.855835 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:38.857202 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:38.858015 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:38.858026 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:38.858608 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:38.859466 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:38.859493 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.861080 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.861087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:38.862709 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:38.862724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.863562 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.863567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:38.863862 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:38.864626 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:38.864657 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:38.866562 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:38.868874 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:38.868884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:38.868904 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:38.869308 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:38.869588 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:38.869594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:38.869599 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:38.869839 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:38.870475 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:38.870485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:38.870850 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 23:54:38.871094 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 23:54:38.871473 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":156,"threads_started":1}
I20260928 23:54:38.874197 22780 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35239
I20260928 23:54:38.874405 22780 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:38.875103 22780 heartbeater.cc:511] Master 127.20.88.62:35239 requested a full tablet report, sending...
I20260928 23:54:38.877072 22576 ts_manager.cc:194] Registered new tserver with Master: 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591)
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:54:38.878762 22576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:37903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:38.885133 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:38.876508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.1:41133)
0928 23:54:38.876836 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:38.876852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:38.876957 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:38.877403 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:38.877406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:38.877655 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:38.877804 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:38.877816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.879061 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.879064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:38.879793 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:38.879804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.880383 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.880387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:38.880482 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:38.881091 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:38.881105 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:38.881558 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:38.883529 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:38.883534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:38.883537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:38.883744 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:38.884041 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:38.884044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:38.884046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:38.884135 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:38.884550 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:38.884554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:38.884656 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 23:54:38.884760 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:54:38.884914 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
I20260928 23:54:38.885125 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:38.876843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:41133 (local address 127.20.88.62:35239)
0928 23:54:38.877002 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:38.877009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:38.877023 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:54:38.877066 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:38.877071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:38.877123 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:38.877231 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:38.877966 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.878807 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.879938 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:38.880197 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:38.880365 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:38.881720 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:38.881726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:38.881732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:38.881767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:38.883413 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.883845 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.883850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.883852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.883942 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:38.884224 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:38.884258 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:38.884260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:38.884451 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:38.884562 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:38.884825 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:54:38.884956 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:38.900743 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35239
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:39.019152 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:39.019465 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:39.019515 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:39.023787 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:39.023865 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:39.023908 22789 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:54:39.023922 22789 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:39.023999 22789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:39.029901 22789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:39.031082 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:39.032163 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:39.042562 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:39.042671 22789 server_base.cc:1034] running on GCE node
W20260928 23:54:39.042532 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:39.042940 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:39.043419 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:39.044139 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:39.045344 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790639679045334 us; error 34 us; skew 500 ppm
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:39.048570 22789 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:39.049603 22789 webserver.cc:460] Webserver started at http://127.20.88.2:40109/ using document root <none> and password file <none>
I20260928 23:54:39.050159 22789 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:39.050220 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:39.050401 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:39.052598 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "4dd1a8638a4b413ba059bc80bf2bc75f"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "2e68b30814a249d6fe60270a4a755e86"
server_key_iv: "ffcbd68399612da74dd4cfa3f5c67d17"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.053123 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "4dd1a8638a4b413ba059bc80bf2bc75f"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "2e68b30814a249d6fe60270a4a755e86"
server_key_iv: "ffcbd68399612da74dd4cfa3f5c67d17"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.057185 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:54:39.064654 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:39.066545 22789 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 23:54:39.066726 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "4dd1a8638a4b413ba059bc80bf2bc75f"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "2e68b30814a249d6fe60270a4a755e86"
server_key_iv: "ffcbd68399612da74dd4cfa3f5c67d17"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.066875 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:39.083233 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:39.086750 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:39.086984 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:39.088901 22789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:39.088980 22789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:39.089042 22789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:39.089088 22789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:39.100646 22789 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:38289
I20260928 23:54:39.100718 22919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:38289 every 8 connection(s)
I20260928 23:54:39.101672 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:39.103524 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.089684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:48229 (local address 127.20.88.62:35239)
0928 23:54:39.089865 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.089870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.090844 (+ 974us) server_negotiation.cc:407] Connection header received
0928 23:54:39.092010 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.092017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.092093 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.092208 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.093840 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.094390 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.095751 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.096099 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.096228 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.099355 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.099360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.099364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.099401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.101142 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.101729 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.101733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.101735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.101863 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.102293 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.102296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.102298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.102549 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.102651 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.103250 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.103401 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:54:39.104526 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.090153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.2:48229)
0928 23:54:39.090657 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.090712 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.091735 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.092450 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.092458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.092900 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:39.093668 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.093686 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.094529 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.094533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.095612 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.095623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.096322 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.096327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.096525 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.097236 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:39.097259 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:39.099137 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:39.101276 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.101290 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.101306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.101582 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.101975 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.101978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.101980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.102166 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.102661 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:39.102667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:39.103019 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.103199 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.103491 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":145,"threads_started":1}
I20260928 23:54:39.105952 22920 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35239
I20260928 23:54:39.106145 22920 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:39.106674 22920 heartbeater.cc:511] Master 127.20.88.62:35239 requested a full tablet report, sending...
I20260928 23:54:39.107805 22577 ts_manager.cc:194] Registered new tserver with Master: 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289)
I20260928 23:54:39.108624 22577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:48229
I20260928 23:54:39.109192 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22789
I20260928 23:54:39.109324 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:39.109625 20832 external_mini_cluster.cc:1428] Setting key 044299223e8863fcd44a0d20605f74ac
I20260928 23:54:39.118321 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.108744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.2:50511)
0928 23:54:39.109219 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.109238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.109402 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.110327 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.110330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.110566 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:39.110686 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.110693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.111526 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.111530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.112600 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.112615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.113117 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.113122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.113241 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.113863 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:39.113876 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:39.114288 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:39.116292 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.116296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.116299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.116549 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.116904 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.116909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.116912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.117013 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.117681 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:39.117689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:39.117809 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.117912 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.118066 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":131,"threads_started":1}
I20260928 23:54:39.118353 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.108878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50511 (local address 127.20.88.62:35239)
0928 23:54:39.110098 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.110105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.110122 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:54:39.110169 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.110172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.110212 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.110305 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.110830 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.111381 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.112745 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.112971 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.113111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.114461 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.114465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.114467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.114493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.116122 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.116689 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.116694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.116695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.116793 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.117172 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.117178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.117181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.117553 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.117669 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.118011 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.118166 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116}
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:39.143465 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35239
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:39.262328 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:39.262671 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:39.262722 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:39.267050 22929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:39.267127 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:39.267168 22929 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:54:39.267184 22929 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:54:39.267216 22929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:39.273191 22929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:39.274379 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:39.275453 22929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:39.285878 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:39.286003 22929 server_base.cc:1034] running on GCE node
W20260928 23:54:39.285830 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:39.285822 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:39.286669 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:39.287341 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:39.288537 22929 hybrid_clock.cc:648] HybridClock initialized: now 1790639679288518 us; error 39 us; skew 500 ppm
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:39.291589 22929 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:39.292732 22929 webserver.cc:460] Webserver started at http://127.20.88.3:40827/ using document root <none> and password file <none>
I20260928 23:54:39.293509 22929 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:39.293624 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:39.293905 22929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:39.296208 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "dff2c7a725e5116e42e964b2d4b050fe"
server_key_iv: "74b56e2617439e9403c06fecf3fdc04c"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.296958 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "dff2c7a725e5116e42e964b2d4b050fe"
server_key_iv: "74b56e2617439e9403c06fecf3fdc04c"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.301245 22929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:54:39.304066 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:39.305611 22929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:39.305774 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
format_stamp: "Formatted at 2026-09-28 23:54:39 on dist-test-slave-wvpw"
server_key: "dff2c7a725e5116e42e964b2d4b050fe"
server_key_iv: "74b56e2617439e9403c06fecf3fdc04c"
server_key_version: "encryptionkey@0"
I20260928 23:54:39.305915 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:39.320653 22929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:39.324487 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:39.324661 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:39.326270 22929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:39.326339 22929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:39.326386 22929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:39.326402 22929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:39.337783 22929 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:43771
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:39.337870 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:43771 every 8 connection(s)
I20260928 23:54:39.338951 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:39.340803 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 22929
I20260928 23:54:39.340946 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:39.341328 20832 external_mini_cluster.cc:1428] Setting key f5d8ed8d0fcf3b4468c34e98fe9a7ad4
I20260928 23:54:39.343114 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.327200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:55673 (local address 127.20.88.62:35239)
0928 23:54:39.327336 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.327340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.328603 (+ 1263us) server_negotiation.cc:407] Connection header received
0928 23:54:39.329875 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.329879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.329949 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.330047 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.331928 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.332671 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.334668 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.334923 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.335022 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.338627 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.338630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.338632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.338664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.340494 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.341124 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.341128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.341131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.341240 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.341715 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.341717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.341719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.341977 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.342094 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.342801 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.342956 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:54:39.344208 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.327742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.3:55673)
0928 23:54:39.328424 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.328472 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.329600 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.330300 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.330314 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.330880 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:39.331707 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.331736 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.332834 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.332841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.334501 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.334512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.335110 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.335115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.335367 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.336397 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:39.336428 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:39.338362 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:39.340638 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.340643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.340657 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.340999 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.341337 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.341339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.341341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.341563 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.342093 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:39.342102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:39.342474 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.342755 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.343173 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":146,"threads_started":1}
I20260928 23:54:39.345587 23060 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35239
I20260928 23:54:39.345762 23060 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:39.346335 23060 heartbeater.cc:511] Master 127.20.88.62:35239 requested a full tablet report, sending...
I20260928 23:54:39.347507 22576 ts_manager.cc:194] Registered new tserver with Master: 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:39.348317 22576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:55673
I20260928 23:54:39.355821 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.347928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44755 (local address 127.20.88.62:35239)
0928 23:54:39.348066 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.348073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.348342 (+ 269us) server_negotiation.cc:407] Connection header received
0928 23:54:39.348459 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.348462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.348505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.348599 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.349157 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.349697 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.350797 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.351065 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.351163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.352502 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.352505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.352507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.352565 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.353857 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.354412 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.354416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.354417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.354524 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.354853 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.354857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.354859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.355114 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.355252 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.355544 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.355690 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:54:39.355875 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.347846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.20.88.3:44755)
0928 23:54:39.348216 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.348275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.348378 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.348615 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.348617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.348848 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:39.348996 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.349004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.349847 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.349850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.350642 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.350655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.351212 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.351215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.351306 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.351907 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:39.351920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:39.352365 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:39.354000 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.354004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.354006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.354279 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.354634 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.354637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.354639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.354748 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.355203 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:39.355206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:39.355323 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.355452 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.355620 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260928 23:54:39.356261 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:39.386092 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.376367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.20.88.62:35239)
0928 23:54:39.376542 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.376546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.376558 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:39.376599 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.376601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.376641 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.376784 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.377592 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.378173 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.379074 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.379318 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.379434 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.382292 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.382296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.382299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.382336 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.383992 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.384596 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.384599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.384601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.384696 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.385022 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.385024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.385025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.385230 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.385344 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.385709 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.385961 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 23:54:39.390177 22576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:39.393553 22576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:39.409138 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.402877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43771 (local address 127.0.0.1:48946)
0928 23:54:39.404044 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.404059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.404215 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.404921 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.404923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.404936 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.405019 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.405023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.406153 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.406160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.407512 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.407522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.408545 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.408549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.408658 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.408700 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.408823 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.408950 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":49,"thread_start_us":115,"threads_started":1}
I20260928 23:54:39.409144 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.403283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.20.88.3:43771)
0928 23:54:39.403903 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.403916 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.404142 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:54:39.404426 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.404431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.404619 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.404748 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.405131 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.406046 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.407632 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.408419 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.408530 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.408703 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.408872 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.408978 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":90,"threads_started":1}
I20260928 23:54:39.409646 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.403792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:46591 (local address 127.0.0.1:35262)
0928 23:54:39.404595 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.404608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.404706 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.405158 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.405160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.405180 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.405266 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.405271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.406592 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.406595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.407837 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.407848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.409094 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.409097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.409290 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.409341 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.409415 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.409501 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":104,"thread_start_us":77,"threads_started":1}
I20260928 23:54:39.409857 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.402805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.0.0.1:53416)
0928 23:54:39.403208 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.403257 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.403379 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.404286 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.404292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.404317 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.404485 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.404492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.406002 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.406007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.407837 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.407848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.409088 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.409092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.409270 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.409323 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.409398 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.409480 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":126,"thread_start_us":84,"threads_started":1}
I20260928 23:54:39.409838 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.402915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.20.88.2:38289)
0928 23:54:39.403715 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.403727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.403750 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:54:39.403827 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.403832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.404105 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.404322 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.404636 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.405838 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.407988 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.408746 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.408841 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.409027 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.409583 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.409682 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":99,"threads_started":1}
I20260928 23:54:39.410140 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.404188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.20.88.1:46591)
0928 23:54:39.404612 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.404619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.404673 (+ 54us) server_negotiation.cc:407] Connection header received
0928 23:54:39.404834 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.404840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.405026 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.405160 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.405384 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.406461 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.407968 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.408777 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.408904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.409032 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.409465 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.409980 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
I20260928 23:54:39.412832 22994 tablet_service.cc:1467] Processing CreateTablet for tablet fff33867fe0245d7b7e85a74efada5be (DEFAULT_TABLE table=test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.412818 22854 tablet_service.cc:1467] Processing CreateTablet for tablet fff33867fe0245d7b7e85a74efada5be (DEFAULT_TABLE table=test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.412817 22714 tablet_service.cc:1467] Processing CreateTablet for tablet fff33867fe0245d7b7e85a74efada5be (DEFAULT_TABLE table=test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.413928 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff33867fe0245d7b7e85a74efada5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.413926 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff33867fe0245d7b7e85a74efada5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.413926 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff33867fe0245d7b7e85a74efada5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.421025 23083 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Bootstrap starting.
I20260928 23:54:39.422888 23085 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Bootstrap starting.
I20260928 23:54:39.424005 23083 tablet_bootstrap.cc:654] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.424608 23084 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Bootstrap starting.
I20260928 23:54:39.424903 23083 log.cc:824] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:39.424949 23085 tablet_bootstrap.cc:654] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.425814 23085 log.cc:824] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:39.426832 23084 tablet_bootstrap.cc:654] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.427095 23083 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: No bootstrap required, opened a new log
I20260928 23:54:39.427331 23083 ts_tablet_manager.cc:1397] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:39.427748 23084 log.cc:824] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:39.428157 23085 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: No bootstrap required, opened a new log
I20260928 23:54:39.428391 23085 ts_tablet_manager.cc:1397] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:54:39.429944 23084 tablet_bootstrap.cc:492] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: No bootstrap required, opened a new log
I20260928 23:54:39.430148 23084 ts_tablet_manager.cc:1397] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:39.430868 23083 raft_consensus.cc:348] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.431142 23083 raft_consensus.cc:374] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.431203 23083 raft_consensus.cc:729] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.431674 23083 consensus_queue.cc:260] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.432438 23060 heartbeater.cc:503] Master 127.20.88.62:35239 was elected leader, sending a full tablet report...
I20260928 23:54:39.432626 23085 raft_consensus.cc:348] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.432839 23083 ts_tablet_manager.cc:1428] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260928 23:54:39.432900 23085 raft_consensus.cc:374] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.432955 23085 raft_consensus.cc:729] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.433483 23085 consensus_queue.cc:260] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.434230 22920 heartbeater.cc:503] Master 127.20.88.62:35239 was elected leader, sending a full tablet report...
I20260928 23:54:39.434564 23085 ts_tablet_manager.cc:1428] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:54:39.434867 23084 raft_consensus.cc:348] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.435070 23084 raft_consensus.cc:374] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.435101 23084 raft_consensus.cc:729] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155ffdae490948b9831d9fea14bf18f8, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.435588 23084 consensus_queue.cc:260] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.436698 23084 ts_tablet_manager.cc:1428] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 23:54:39.436914 22780 heartbeater.cc:503] Master 127.20.88.62:35239 was elected leader, sending a full tablet report...
I20260928 23:54:39.586340 23089 raft_consensus.cc:482] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:39.586604 23089 raft_consensus.cc:504] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.588064 23089 leader_election.cc:288] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289), 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591)
W20260928 23:54:39.590003 23061 tablet.cc:2369] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:39.592473 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.588444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44439 (local address 127.20.88.2:38289)
0928 23:54:39.588618 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.588622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.588641 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:54:39.588729 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.588733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.588796 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.588882 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.589209 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.589761 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.591329 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.591757 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.591926 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.592009 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.592255 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.592340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 23:54:39.592473 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.588283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.20.88.3:44439)
0928 23:54:39.588467 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.588486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.588615 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.588912 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.588914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.588935 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.589045 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.589050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.589899 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.589906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.591183 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.591194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.591923 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.591927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.592069 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.592091 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.592208 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.592318 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 23:54:39.593211 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff33867fe0245d7b7e85a74efada5be" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" is_pre_election: true
I20260928 23:54:39.593327 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.588488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:46591 (local address 127.20.88.3:49905)
0928 23:54:39.589000 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.589014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.589132 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.589409 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.589412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.589427 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.589979 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.589989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.591057 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.591060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.591919 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.591928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.592854 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.592860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.592966 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.592993 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.593095 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.593166 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":158,"threads_started":1}
I20260928 23:54:39.593321 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.588643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:49905 (local address 127.20.88.1:46591)
0928 23:54:39.588762 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.588767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.589103 (+ 336us) server_negotiation.cc:407] Connection header received
0928 23:54:39.589254 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.589261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.589308 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.589401 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.590128 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.590893 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.592102 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.592725 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.592823 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.592886 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.593139 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.593211 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:54:39.593590 22874 raft_consensus.cc:2454] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 0.
I20260928 23:54:39.593950 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff33867fe0245d7b7e85a74efada5be" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8" is_pre_election: true
I20260928 23:54:39.594188 22945 leader_election.cc:302] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd1a8638a4b413ba059bc80bf2bc75f, 635a344b9bfd42dea0edc1c1215a2c5a; no voters:
I20260928 23:54:39.594318 22734 raft_consensus.cc:2454] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 0.
I20260928 23:54:39.594537 23089 raft_consensus.cc:2781] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:39.594661 23089 raft_consensus.cc:482] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:39.594738 23089 raft_consensus.cc:3037] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.595923 23089 raft_consensus.cc:504] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
I20260928 23:54:39.596400 23089 leader_election.cc:288] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 election: Requested vote from peers 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289), 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591)
I20260928 23:54:39.596746 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff33867fe0245d7b7e85a74efada5be" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8"
I20260928 23:54:39.596820 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff33867fe0245d7b7e85a74efada5be" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f"
I20260928 23:54:39.596891 22734 raft_consensus.cc:3037] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.597000 22874 raft_consensus.cc:3037] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.598033 22734 raft_consensus.cc:2454] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 1.
I20260928 23:54:39.598098 22874 raft_consensus.cc:2454] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 1.
I20260928 23:54:39.598469 22946 leader_election.cc:302] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f, 635a344b9bfd42dea0edc1c1215a2c5a; no voters:
I20260928 23:54:39.598654 23089 raft_consensus.cc:2781] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:39.598908 23089 raft_consensus.cc:686] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 LEADER]: Becoming Leader. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Running, Role: LEADER
I20260928 23:54:39.599327 23089 consensus_queue.cc:237] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } }
W20260928 23:54:39.602952 22921 tablet.cc:2369] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:39.603013 22575 catalog_manager.cc:5495] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3). New cstate: current_term: 1 leader_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } health_report { overall_health: UNKNOWN } } }
W20260928 23:54:39.617727 22781 tablet.cc:2369] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 23:54:39 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:39.679631 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.670272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.20.88.62:35239)
0928 23:54:39.670421 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.670427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.670724 (+ 297us) server_negotiation.cc:407] Connection header received
0928 23:54:39.670818 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.670821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.670864 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.670979 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:39.671686 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.672273 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.673195 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.673417 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.673553 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.676187 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:39.676190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:39.676192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:39.676214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:39.677789 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.678301 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.678306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.678308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.678384 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:39.678659 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:39.678662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:39.678664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:39.678864 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:39.678980 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.679281 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.679502 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":52}
I20260928 23:54:39.679631 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.670240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35239 (local address 127.0.0.1:32902)
0928 23:54:39.670613 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.670627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.670733 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.671010 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.671012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.671437 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:39.671550 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.671556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.672364 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.672366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.673082 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.673092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.673586 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.673589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.673681 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.674410 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:39.674423 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:39.676023 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:39.677920 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.677933 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.677940 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.678184 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.678477 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:39.678481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:39.678483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:39.678567 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:39.678980 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:39.678989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:39.679109 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.679235 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.679427 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":3,"thread_start_us":159,"threads_started":1}
I20260928 23:54:39.684188 22575 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:32902:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:54:39.685035 22575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:39.690562 22994 tablet_service.cc:1467] Processing CreateTablet for tablet 17e1d43e99844fe6b595b0433b977e42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39f759be48134f5d8ead14b1522dff86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:39.690387 22714 tablet_service.cc:1467] Processing CreateTablet for tablet 17e1d43e99844fe6b595b0433b977e42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39f759be48134f5d8ead14b1522dff86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:39.690398 22854 tablet_service.cc:1467] Processing CreateTablet for tablet 17e1d43e99844fe6b595b0433b977e42 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=39f759be48134f5d8ead14b1522dff86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:54:39.691223 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e1d43e99844fe6b595b0433b977e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.691223 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e1d43e99844fe6b595b0433b977e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.691228 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17e1d43e99844fe6b595b0433b977e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.694279 23084 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: Bootstrap starting.
I20260928 23:54:39.694299 23085 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: Bootstrap starting.
I20260928 23:54:39.695277 23084 tablet_bootstrap.cc:654] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.695279 23085 tablet_bootstrap.cc:654] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.696038 23083 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: Bootstrap starting.
I20260928 23:54:39.696693 23084 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: No bootstrap required, opened a new log
I20260928 23:54:39.696816 23084 ts_tablet_manager.cc:1397] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:39.697319 23083 tablet_bootstrap.cc:654] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.697517 23084 raft_consensus.cc:348] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.697656 23084 raft_consensus.cc:374] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.697700 23084 raft_consensus.cc:729] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155ffdae490948b9831d9fea14bf18f8, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.697852 23084 consensus_queue.cc:260] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.698156 23084 ts_tablet_manager.cc:1428] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:39.698503 23085 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: No bootstrap required, opened a new log
I20260928 23:54:39.698589 23085 ts_tablet_manager.cc:1397] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:54:39.699112 23085 raft_consensus.cc:348] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.699232 23085 raft_consensus.cc:374] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.699258 23085 raft_consensus.cc:729] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.699399 23085 consensus_queue.cc:260] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.699672 23090 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.700022 23090 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.700011 23091 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.700165 23091 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.700783 23083 tablet_bootstrap.cc:492] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: No bootstrap required, opened a new log
I20260928 23:54:39.700917 23083 ts_tablet_manager.cc:1397] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 23:54:39.701439 23083 raft_consensus.cc:348] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.701584 23083 raft_consensus.cc:374] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.701617 23083 raft_consensus.cc:729] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.701748 23083 consensus_queue.cc:260] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.702051 23083 ts_tablet_manager.cc:1428] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:39.702008 23089 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.702136 23089 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.699693 23085 ts_tablet_manager.cc:1428] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:39.816154 23090 raft_consensus.cc:482] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:39.816411 23090 raft_consensus.cc:504] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.817683 23090 leader_election.cc:288] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:39.822168 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.818031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:46591 (local address 127.20.88.2:60171)
0928 23:54:39.818315 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.818331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.818500 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.818825 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.818828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.818849 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.818971 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.818976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.819731 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.819738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.821032 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.821042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.821816 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.821820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.821896 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.821919 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.821978 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.822035 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":27,"thread_start_us":82,"threads_started":1}
I20260928 23:54:39.822230 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.818212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:60171 (local address 127.20.88.1:46591)
0928 23:54:39.818387 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.818392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.818436 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:54:39.818642 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.818646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.818699 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.818777 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.819092 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.819610 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.821171 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.821660 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.821773 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.821843 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.822015 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.822099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 23:54:39.822788 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e1d43e99844fe6b595b0433b977e42" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8" is_pre_election: true
I20260928 23:54:39.822921 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.818426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43771 (local address 127.20.88.2:41549)
0928 23:54:39.818758 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.818770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.818922 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.819180 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.819183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.819198 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:39.819672 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.819679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.820545 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.820548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.821478 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.821485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.822457 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.822463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.822573 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.822599 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.822696 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.822759 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1}
I20260928 23:54:39.822938 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.818562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:41549 (local address 127.20.88.3:43771)
0928 23:54:39.818699 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.818704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.818862 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:54:39.819029 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.819032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.819082 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.819167 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:39.819797 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.820452 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.821599 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.822214 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.822328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.822392 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.822739 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.822816 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:54:39.822959 22734 raft_consensus.cc:2454] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:39.823385 22806 leader_election.cc:302] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f; no voters:
I20260928 23:54:39.823696 23090 raft_consensus.cc:2781] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:39.823626 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e1d43e99844fe6b595b0433b977e42" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" is_pre_election: true
I20260928 23:54:39.823776 23090 raft_consensus.cc:482] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:39.823818 23090 raft_consensus.cc:3037] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.823894 23014 raft_consensus.cc:2454] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:39.824985 23090 raft_consensus.cc:504] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.825307 23090 leader_election.cc:288] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Requested vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:39.825695 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e1d43e99844fe6b595b0433b977e42" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8"
I20260928 23:54:39.825765 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17e1d43e99844fe6b595b0433b977e42" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
I20260928 23:54:39.825893 22734 raft_consensus.cc:3037] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.825893 23014 raft_consensus.cc:3037] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:39.827126 23014 raft_consensus.cc:2454] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:39.827127 22734 raft_consensus.cc:2454] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:39.827438 22806 leader_election.cc:302] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f; no voters:
I20260928 23:54:39.827621 23090 raft_consensus.cc:2781] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:39.827879 23090 raft_consensus.cc:686] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 LEADER]: Becoming Leader. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Running, Role: LEADER
I20260928 23:54:39.828311 23090 consensus_queue.cc:237] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.830849 23106 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: TxnStatusTablet state changed. Reason: New leader 4dd1a8638a4b413ba059bc80bf2bc75f. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.830978 23106 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:39.831295 23108 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:54:39.831513 23108 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:54:39.832180 22576 catalog_manager.cc:5495] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2). New cstate: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } health_report { overall_health: UNKNOWN } } }
I20260928 23:54:39.862515 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.858156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.0.0.1:53422)
0928 23:54:39.858306 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.858324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.858442 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.858864 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.858866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.858872 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:39.858972 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.858977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.860020 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.860023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.860701 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.860712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.861293 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.861295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.861364 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.861413 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.862239 (+ 826us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.862252 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.862310 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.862397 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:54:39.862527 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.858274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.20.88.2:38289)
0928 23:54:39.858434 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.858438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.858457 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:54:39.858616 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.858619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.858689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.858809 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:39.859085 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.859880 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.860819 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.861169 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.861280 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.861551 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.862142 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.862268 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.862324 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.862384 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 23:54:39.868933 23014 raft_consensus.cc:1261] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:39.869063 22734 raft_consensus.cc:1261] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:39.869712 23106 consensus_queue.cc:1035] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:39.870007 23090 consensus_queue.cc:1035] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:39.874010 23091 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: TxnStatusTablet state changed. Reason: New leader 4dd1a8638a4b413ba059bc80bf2bc75f. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.874143 23091 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.874140 23089 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: TxnStatusTablet state changed. Reason: New leader 4dd1a8638a4b413ba059bc80bf2bc75f. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.874269 23089 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.879041 23111 mvcc.cc:205] Tried to move back new op lower bound from 7334460128735141888 to 7334460128579928064. Current Snapshot: MvccSnapshot[applied={T|T < 7334460128735141888}]
I20260928 23:54:39.879262 23089 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.879396 23089 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 635a344b9bfd42dea0edc1c1215a2c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.879789 23106 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.879913 23106 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:39.880640 23112 mvcc.cc:205] Tried to move back new op lower bound from 7334460128735141888 to 7334460128579928064. Current Snapshot: MvccSnapshot[applied={T|T < 7334460128735141888}]
I20260928 23:54:39.880970 23090 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.881065 23090 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 4dd1a8638a4b413ba059bc80bf2bc75f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:54:39.881174 23113 mvcc.cc:205] Tried to move back new op lower bound from 7334460128735141888 to 7334460128579928064. Current Snapshot: MvccSnapshot[applied={T|T < 7334460128735141888}]
I20260928 23:54:39.881299 23091 tablet_replica.cc:439] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } } }
I20260928 23:54:39.881409 23091 tablet_replica.cc:442] T 17e1d43e99844fe6b595b0433b977e42 P 155ffdae490948b9831d9fea14bf18f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:54:39.892204 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.887810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.20.88.3:43771)
0928 23:54:39.887952 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.887957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.888181 (+ 224us) server_negotiation.cc:407] Connection header received
0928 23:54:39.888372 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.888374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.888433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.888528 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:39.888882 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.889405 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.890716 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.890938 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.891020 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.891318 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.891781 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.891865 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.891983 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.892035 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:54:39.904773 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.900946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.20.88.3:50353)
0928 23:54:39.901400 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.901412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.901555 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.901829 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.901833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.901840 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:39.901969 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.901975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.902724 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.902728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.903394 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.903403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.903889 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.903894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.903988 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.904039 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.904441 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.904450 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.904531 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.904625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":105,"threads_started":1}
I20260928 23:54:39.904759 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.901073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:50353 (local address 127.20.88.2:38289)
0928 23:54:39.901226 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.901232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.901491 (+ 259us) server_negotiation.cc:407] Connection header received
0928 23:54:39.901664 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.901669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.901719 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.901819 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:39.902075 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.902601 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.903529 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.903754 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.903875 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.904218 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.904352 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.904434 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.904553 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.904605 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:54:39.913786 22734 raft_consensus.cc:1261] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Refusing update from remote peer 635a344b9bfd42dea0edc1c1215a2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:39.913858 22874 raft_consensus.cc:1261] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Refusing update from remote peer 635a344b9bfd42dea0edc1c1215a2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:39.914315 23089 consensus_queue.cc:1035] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:39.914503 23093 consensus_queue.cc:1035] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:39.942569 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.938373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:52971 (local address 127.20.88.3:43771)
0928 23:54:39.938500 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:39.938505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:39.938851 (+ 346us) server_negotiation.cc:407] Connection header received
0928 23:54:39.939004 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:39.939008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:39.939078 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:39.939184 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:39.939496 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.940284 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.941232 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.941544 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.941656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.942031 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.942125 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.942196 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:39.942331 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:54:39.942396 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:54:39.942587 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:39.938082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43771 (local address 127.20.88.2:52971)
0928 23:54:39.938756 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:39.938773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:39.938885 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:39.939196 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:39.939199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:39.939208 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:39.939374 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.939380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.940381 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.940385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.941105 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:39.941116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:39.941660 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:39.941664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:39.941835 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:39.941896 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:39.942236 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:39.942244 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:54:39.942317 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:54:39.942402 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":92,"threads_started":1}
I20260928 23:54:39.970713 22576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:39.970940 22576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:39.973961 22576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Sending DeleteTablet for 3 replicas of tablet fff33867fe0245d7b7e85a74efada5be
I20260928 23:54:39.974788 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet fff33867fe0245d7b7e85a74efada5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:39 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:53416
I20260928 23:54:39.975003 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet fff33867fe0245d7b7e85a74efada5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:39 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260928 23:54:39.975617 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet fff33867fe0245d7b7e85a74efada5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:39 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:35262
I20260928 23:54:39.976060 23136 tablet_replica.cc:330] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: stopping tablet replica
I20260928 23:54:39.976405 23135 tablet_replica.cc:330] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: stopping tablet replica
I20260928 23:54:39.976361 22576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:54:39.976704 23135 raft_consensus.cc:2229] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 LEADER]: Raft consensus shutting down.
W20260928 23:54:39.976795 22576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:39.977100 23135 raft_consensus.cc:2258] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:39.977506 23136 raft_consensus.cc:2229] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:39.977717 23136 raft_consensus.cc:2258] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:39.979182 23136 ts_tablet_manager.cc:1905] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:39.980897 23134 tablet_replica.cc:330] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: stopping tablet replica
I20260928 23:54:39.981273 23134 raft_consensus.cc:2229] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:39.981523 23134 raft_consensus.cc:2258] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:39.982406 22714 tablet_service.cc:1467] Processing CreateTablet for tablet df5575abec30481bb541a71bf6b5eb88 (DEFAULT_TABLE table=test-table [id=e1463f49910840b9ade71cbc3132df90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.982671 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5575abec30481bb541a71bf6b5eb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.982678 22854 tablet_service.cc:1467] Processing CreateTablet for tablet df5575abec30481bb541a71bf6b5eb88 (DEFAULT_TABLE table=test-table [id=e1463f49910840b9ade71cbc3132df90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.982926 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5575abec30481bb541a71bf6b5eb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.984820 22994 tablet_service.cc:1467] Processing CreateTablet for tablet df5575abec30481bb541a71bf6b5eb88 (DEFAULT_TABLE table=test-table [id=e1463f49910840b9ade71cbc3132df90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:39.985102 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5575abec30481bb541a71bf6b5eb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:39.985536 23085 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Bootstrap starting.
I20260928 23:54:39.986506 23085 tablet_bootstrap.cc:654] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.987627 23083 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Bootstrap starting.
I20260928 23:54:39.988598 23083 tablet_bootstrap.cc:654] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.989279 23134 ts_tablet_manager.cc:1905] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:39.990386 23084 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Bootstrap starting.
I20260928 23:54:39.991348 23084 tablet_bootstrap.cc:654] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:39.993292 23135 ts_tablet_manager.cc:1905] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:39.993708 23085 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: No bootstrap required, opened a new log
I20260928 23:54:39.993800 23085 ts_tablet_manager.cc:1397] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 23:54:39.994222 23085 raft_consensus.cc:348] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.994328 23085 raft_consensus.cc:374] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.994356 23085 raft_consensus.cc:729] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.994472 23085 consensus_queue.cc:260] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.994709 23085 ts_tablet_manager.cc:1428] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:39.996130 23083 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: No bootstrap required, opened a new log
I20260928 23:54:39.996209 23083 ts_tablet_manager.cc:1397] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260928 23:54:39.996641 23083 raft_consensus.cc:348] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.996742 23083 raft_consensus.cc:374] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:39.996767 23083 raft_consensus.cc:729] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Initialized, Role: FOLLOWER
I20260928 23:54:39.996879 23083 consensus_queue.cc:260] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:39.997119 23083 ts_tablet_manager.cc:1428] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:39.999053 23134 ts_tablet_manager.cc:1918] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:39.999207 23134 log.cc:1196] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/fff33867fe0245d7b7e85a74efada5be
I20260928 23:54:39.999614 23134 ts_tablet_manager.cc:1939] T fff33867fe0245d7b7e85a74efada5be P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting consensus metadata
I20260928 23:54:40.001097 23135 ts_tablet_manager.cc:1918] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:40.001194 23135 log.cc:1196] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/fff33867fe0245d7b7e85a74efada5be
I20260928 23:54:40.001498 23135 ts_tablet_manager.cc:1939] T fff33867fe0245d7b7e85a74efada5be P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting consensus metadata
I20260928 23:54:40.002164 22561 catalog_manager.cc:4841] TS 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289): tablet fff33867fe0245d7b7e85a74efada5be (table test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]) successfully deleted
I20260928 23:54:40.002512 22562 catalog_manager.cc:4841] TS 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771): tablet fff33867fe0245d7b7e85a74efada5be (table test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]) successfully deleted
I20260928 23:54:40.002982 23084 tablet_bootstrap.cc:492] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: No bootstrap required, opened a new log
I20260928 23:54:40.003087 23084 ts_tablet_manager.cc:1397] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 23:54:40.003554 23084 raft_consensus.cc:348] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:40.003680 23084 raft_consensus.cc:374] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:40.003716 23084 raft_consensus.cc:729] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155ffdae490948b9831d9fea14bf18f8, State: Initialized, Role: FOLLOWER
I20260928 23:54:40.003835 23084 consensus_queue.cc:260] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:40.004110 23084 ts_tablet_manager.cc:1428] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:40.005548 23136 ts_tablet_manager.cc:1918] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:40.006115 23136 log.cc:1196] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/fff33867fe0245d7b7e85a74efada5be
I20260928 23:54:40.006567 23136 ts_tablet_manager.cc:1939] T fff33867fe0245d7b7e85a74efada5be P 155ffdae490948b9831d9fea14bf18f8: Deleting consensus metadata
I20260928 23:54:40.007925 22562 catalog_manager.cc:4841] TS 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591): tablet fff33867fe0245d7b7e85a74efada5be (table test-table [id=6a15e2e0d4734578b8e30d2f3a12e24f]) successfully deleted
I20260928 23:54:40.118618 23116 raft_consensus.cc:482] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:40.118960 23116 raft_consensus.cc:504] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:40.119947 23116 leader_election.cc:288] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:40.120389 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5575abec30481bb541a71bf6b5eb88" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8" is_pre_election: true
I20260928 23:54:40.120529 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5575abec30481bb541a71bf6b5eb88" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" is_pre_election: true
I20260928 23:54:40.120679 23014 raft_consensus.cc:2454] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:40.120661 22734 raft_consensus.cc:2454] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:40.121333 22806 leader_election.cc:302] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f; no voters:
I20260928 23:54:40.121628 23116 raft_consensus.cc:2781] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:40.121697 23116 raft_consensus.cc:482] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:40.121734 23116 raft_consensus.cc:3037] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:40.123101 23116 raft_consensus.cc:504] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:40.123543 23116 leader_election.cc:288] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Requested vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:40.124035 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5575abec30481bb541a71bf6b5eb88" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8"
I20260928 23:54:40.124172 22734 raft_consensus.cc:3037] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:40.124383 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5575abec30481bb541a71bf6b5eb88" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
I20260928 23:54:40.124524 23014 raft_consensus.cc:3037] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:40.125404 22734 raft_consensus.cc:2454] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:40.125521 23014 raft_consensus.cc:2454] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:40.125860 22806 leader_election.cc:302] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f; no voters:
I20260928 23:54:40.126359 23116 raft_consensus.cc:2781] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:40.126483 23116 raft_consensus.cc:686] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 LEADER]: Becoming Leader. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Running, Role: LEADER
I20260928 23:54:40.126631 23116 consensus_queue.cc:237] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:40.128806 22576 catalog_manager.cc:5495] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2). New cstate: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } health_report { overall_health: UNKNOWN } } }
I20260928 23:54:40.157312 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:40.154050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.20.88.2:38289)
0928 23:54:40.154179 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:40.154183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:40.154196 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:54:40.154316 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:40.154319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:40.154362 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:40.154429 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:40.154700 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.155342 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.156339 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.156566 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.156656 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:40.156895 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:40.156972 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:40.157026 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:40.157119 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:54:40.157167 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:54:40.167683 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:40.164295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.20.88.2:46209)
0928 23:54:40.164486 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:40.164502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:40.164628 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:40.164877 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:40.164880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:40.164886 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:54:40.165067 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:40.165073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.165788 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.165794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:40.166480 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:40.166492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.166968 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.166970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:40.167036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:40.167062 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:54:40.167388 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:54:40.167395 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:54:40.167464 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:54:40.167543 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":33}
I20260928 23:54:40.167703 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:40.164360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:46209 (local address 127.20.88.2:38289)
0928 23:54:40.164480 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:40.164485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:40.164574 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:54:40.164727 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:40.164730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:40.164780 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:40.164873 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:40.165169 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.165676 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.166624 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:40.166856 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:40.166945 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:40.167183 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:40.167258 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:40.167337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:40.167516 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:54:40.167565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":41}
I20260928 23:54:40.173780 22734 raft_consensus.cc:1261] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:40.173851 23014 raft_consensus.cc:1261] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:40.174285 23118 consensus_queue.cc:1035] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:40.174499 23116 consensus_queue.cc:1035] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:54:40.189718 23111 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:54:41.748931 22577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:41.749095 22577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:41.751848 22577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Sending DeleteTablet for 3 replicas of tablet df5575abec30481bb541a71bf6b5eb88
I20260928 23:54:41.752537 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet df5575abec30481bb541a71bf6b5eb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:41 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:53416
I20260928 23:54:41.752521 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet df5575abec30481bb541a71bf6b5eb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:41 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:35262
I20260928 23:54:41.752585 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet df5575abec30481bb541a71bf6b5eb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:41 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260928 23:54:41.752944 23165 tablet_replica.cc:330] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: stopping tablet replica
I20260928 23:54:41.753222 23165 raft_consensus.cc:2229] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:41.753578 23165 raft_consensus.cc:2258] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:41.753878 23164 tablet_replica.cc:330] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: stopping tablet replica
I20260928 23:54:41.754060 23164 raft_consensus.cc:2229] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:41.754196 23164 raft_consensus.cc:2258] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:41.754791 23165 ts_tablet_manager.cc:1905] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:41.754938 23164 ts_tablet_manager.cc:1905] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:41.757230 23165 ts_tablet_manager.cc:1918] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:54:41.757344 23165 log.cc:1196] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/df5575abec30481bb541a71bf6b5eb88
I20260928 23:54:41.757504 23166 tablet_replica.cc:330] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: stopping tablet replica
I20260928 23:54:41.757637 23166 raft_consensus.cc:2229] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:41.757676 23165 ts_tablet_manager.cc:1939] T df5575abec30481bb541a71bf6b5eb88 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting consensus metadata
I20260928 23:54:41.757791 23166 raft_consensus.cc:2258] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:41.758046 23164 ts_tablet_manager.cc:1918] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:54:41.758127 23164 log.cc:1196] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/df5575abec30481bb541a71bf6b5eb88
I20260928 23:54:41.758419 23164 ts_tablet_manager.cc:1939] T df5575abec30481bb541a71bf6b5eb88 P 155ffdae490948b9831d9fea14bf18f8: Deleting consensus metadata
I20260928 23:54:41.758498 22561 catalog_manager.cc:4841] TS 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289): tablet df5575abec30481bb541a71bf6b5eb88 (table test-table [id=e1463f49910840b9ade71cbc3132df90]) successfully deleted
I20260928 23:54:41.759268 22562 catalog_manager.cc:4841] TS 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591): tablet df5575abec30481bb541a71bf6b5eb88 (table test-table [id=e1463f49910840b9ade71cbc3132df90]) successfully deleted
I20260928 23:54:41.761153 23166 ts_tablet_manager.cc:1905] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:41.763423 23166 ts_tablet_manager.cc:1918] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:54:41.763531 23166 log.cc:1196] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/df5575abec30481bb541a71bf6b5eb88
I20260928 23:54:41.763780 23166 ts_tablet_manager.cc:1939] T df5575abec30481bb541a71bf6b5eb88 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting consensus metadata
I20260928 23:54:41.764650 22562 catalog_manager.cc:4841] TS 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771): tablet df5575abec30481bb541a71bf6b5eb88 (table test-table [id=e1463f49910840b9ade71cbc3132df90]) successfully deleted
Sep 28 23:54:41 dist-test-slave-wvpw krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:54:41 dist-test-slave-wvpw krb5kdc[22532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639681, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:41.788676 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:41.775233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.20.88.62:35239)
0928 23:54:41.775572 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:41.775579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:41.775595 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:54:41.775689 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:41.775692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:41.775737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:41.775818 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:41.776588 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:41.777162 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:41.778026 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:41.778254 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:41.778333 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:41.784860 (+ 6527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:41.784864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:41.784866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:41.784902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:41.786601 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:41.787180 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:41.787184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:41.787186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:41.787274 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:41.787576 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:41.787579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:41.787581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:41.787844 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:41.788063 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:41.788327 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 23:54:41.788452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260928 23:54:41.791354 22577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:41.791839 22577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:41.796574 22714 tablet_service.cc:1467] Processing CreateTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 (DEFAULT_TABLE table=test-table [id=801931c0dd7a4971b13147e29a87361b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:41.796629 22854 tablet_service.cc:1467] Processing CreateTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 (DEFAULT_TABLE table=test-table [id=801931c0dd7a4971b13147e29a87361b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:41.796793 22994 tablet_service.cc:1467] Processing CreateTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 (DEFAULT_TABLE table=test-table [id=801931c0dd7a4971b13147e29a87361b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:41.796936 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14125ec9fad4fbbb6104c0cdbea8d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:41.796934 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14125ec9fad4fbbb6104c0cdbea8d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:41.797053 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14125ec9fad4fbbb6104c0cdbea8d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:41.799029 23175 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Bootstrap starting.
I20260928 23:54:41.799463 23176 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Bootstrap starting.
I20260928 23:54:41.799950 23177 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Bootstrap starting.
I20260928 23:54:41.800173 23175 tablet_bootstrap.cc:654] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:41.800596 23176 tablet_bootstrap.cc:654] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:41.800870 23177 tablet_bootstrap.cc:654] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:41.801586 23175 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: No bootstrap required, opened a new log
I20260928 23:54:41.801676 23175 ts_tablet_manager.cc:1397] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:41.801837 23176 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: No bootstrap required, opened a new log
I20260928 23:54:41.801924 23176 ts_tablet_manager.cc:1397] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:41.802024 23177 tablet_bootstrap.cc:492] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: No bootstrap required, opened a new log
I20260928 23:54:41.802088 23177 ts_tablet_manager.cc:1397] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:54:41.802340 23176 raft_consensus.cc:348] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.802368 23175 raft_consensus.cc:348] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.802402 23177 raft_consensus.cc:348] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.802459 23176 raft_consensus.cc:374] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:41.802459 23175 raft_consensus.cc:374] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:41.802484 23177 raft_consensus.cc:374] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:41.802503 23177 raft_consensus.cc:729] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Initialized, Role: FOLLOWER
I20260928 23:54:41.802493 23176 raft_consensus.cc:729] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Initialized, Role: FOLLOWER
I20260928 23:54:41.802493 23175 raft_consensus.cc:729] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155ffdae490948b9831d9fea14bf18f8, State: Initialized, Role: FOLLOWER
I20260928 23:54:41.802644 23175 consensus_queue.cc:260] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.802644 23176 consensus_queue.cc:260] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.802646 23177 consensus_queue.cc:260] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.803001 23176 ts_tablet_manager.cc:1428] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:41.803000 23175 ts_tablet_manager.cc:1428] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:41.803076 23177 ts_tablet_manager.cc:1428] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:41.923630 23182 raft_consensus.cc:482] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:41.923830 23182 raft_consensus.cc:504] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.924440 23182 leader_election.cc:288] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289)
I20260928 23:54:41.924947 22733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14125ec9fad4fbbb6104c0cdbea8d10" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8" is_pre_election: true
I20260928 23:54:41.924959 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14125ec9fad4fbbb6104c0cdbea8d10" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" is_pre_election: true
I20260928 23:54:41.925117 22733 raft_consensus.cc:2454] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 0.
I20260928 23:54:41.925132 22874 raft_consensus.cc:2454] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 0.
I20260928 23:54:41.925544 22945 leader_election.cc:302] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd1a8638a4b413ba059bc80bf2bc75f, 635a344b9bfd42dea0edc1c1215a2c5a; no voters:
I20260928 23:54:41.925796 23182 raft_consensus.cc:2781] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:41.925860 23182 raft_consensus.cc:482] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:41.925901 23182 raft_consensus.cc:3037] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:41.927052 23182 raft_consensus.cc:504] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.927479 23182 leader_election.cc:288] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 election: Requested vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289)
I20260928 23:54:41.927870 22733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14125ec9fad4fbbb6104c0cdbea8d10" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8"
I20260928 23:54:41.927933 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14125ec9fad4fbbb6104c0cdbea8d10" candidate_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f"
I20260928 23:54:41.928018 22733 raft_consensus.cc:3037] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:41.928056 22874 raft_consensus.cc:3037] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:41.929065 22874 raft_consensus.cc:2454] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 1.
I20260928 23:54:41.929148 22733 raft_consensus.cc:2454] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635a344b9bfd42dea0edc1c1215a2c5a in term 1.
I20260928 23:54:41.929404 22945 leader_election.cc:302] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd1a8638a4b413ba059bc80bf2bc75f, 635a344b9bfd42dea0edc1c1215a2c5a; no voters:
I20260928 23:54:41.929625 23182 raft_consensus.cc:2781] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:41.929869 23182 raft_consensus.cc:686] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 LEADER]: Becoming Leader. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Running, Role: LEADER
I20260928 23:54:41.930038 23182 consensus_queue.cc:237] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:41.931825 22575 catalog_manager.cc:5495] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3). New cstate: current_term: 1 leader_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } health_report { overall_health: HEALTHY } } }
I20260928 23:54:41.968950 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:41.965797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.20.88.3:43771)
0928 23:54:41.966083 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:41.966090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:41.966105 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:54:41.966148 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:41.966150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:41.966197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:41.966279 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:41.966504 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:41.967089 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:41.967929 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:41.968164 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:41.968282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:41.968524 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:41.968601 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:41.968699 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:41.968730 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 23:54:41.968778 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
I20260928 23:54:41.975726 22874 raft_consensus.cc:1261] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Refusing update from remote peer 635a344b9bfd42dea0edc1c1215a2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:41.976174 22733 raft_consensus.cc:1261] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Refusing update from remote peer 635a344b9bfd42dea0edc1c1215a2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:41.976272 23182 consensus_queue.cc:1035] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:41.976595 23182 consensus_queue.cc:1035] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:42.008514 22575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:42.008713 22575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:42.011045 22575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Sending DeleteTablet for 3 replicas of tablet c14125ec9fad4fbbb6104c0cdbea8d10
I20260928 23:54:42.011696 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:53416
I20260928 23:54:42.011932 23165 tablet_replica.cc:330] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: stopping tablet replica
I20260928 23:54:42.011994 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:35262
I20260928 23:54:42.012086 23165 raft_consensus.cc:2229] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:42.012161 23164 tablet_replica.cc:330] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: stopping tablet replica
I20260928 23:54:42.012212 23165 raft_consensus.cc:2258] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:42.012260 23164 raft_consensus.cc:2229] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:42.012406 23164 raft_consensus.cc:2258] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:42.012476 22575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:54:42.012667 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet c14125ec9fad4fbbb6104c0cdbea8d10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260928 23:54:42.012861 23166 tablet_replica.cc:330] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: stopping tablet replica
W20260928 23:54:42.012960 22575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:42.013006 23166 raft_consensus.cc:2229] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:42.013115 23164 ts_tablet_manager.cc:1905] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:42.013239 23166 raft_consensus.cc:2258] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:42.013942 23165 ts_tablet_manager.cc:1905] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:42.015910 23164 ts_tablet_manager.cc:1918] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:42.016012 23164 log.cc:1196] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/c14125ec9fad4fbbb6104c0cdbea8d10
I20260928 23:54:42.016340 23164 ts_tablet_manager.cc:1939] T c14125ec9fad4fbbb6104c0cdbea8d10 P 155ffdae490948b9831d9fea14bf18f8: Deleting consensus metadata
I20260928 23:54:42.016860 23166 ts_tablet_manager.cc:1905] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:42.017092 22562 catalog_manager.cc:4841] TS 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591): tablet c14125ec9fad4fbbb6104c0cdbea8d10 (table test-table [id=801931c0dd7a4971b13147e29a87361b]) successfully deleted
I20260928 23:54:42.018594 22854 tablet_service.cc:1467] Processing CreateTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 (DEFAULT_TABLE table=test-table [id=7076f4b6cb264b3b9bacc189ace80414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:42.018890 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de28d9c5c1324b56a4672b7ecbee5297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:42.019577 23166 ts_tablet_manager.cc:1918] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:42.019662 23166 log.cc:1196] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/c14125ec9fad4fbbb6104c0cdbea8d10
I20260928 23:54:42.019927 23166 ts_tablet_manager.cc:1939] T c14125ec9fad4fbbb6104c0cdbea8d10 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting consensus metadata
I20260928 23:54:42.020521 22714 tablet_service.cc:1467] Processing CreateTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 (DEFAULT_TABLE table=test-table [id=7076f4b6cb264b3b9bacc189ace80414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:42.020550 22994 tablet_service.cc:1467] Processing CreateTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 (DEFAULT_TABLE table=test-table [id=7076f4b6cb264b3b9bacc189ace80414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:42.020799 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de28d9c5c1324b56a4672b7ecbee5297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:42.020821 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de28d9c5c1324b56a4672b7ecbee5297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:42.021502 23176 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Bootstrap starting.
I20260928 23:54:42.022473 23176 tablet_bootstrap.cc:654] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:42.022872 23175 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8: Bootstrap starting.
I20260928 23:54:42.023681 23165 ts_tablet_manager.cc:1918] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:54:42.023708 22562 catalog_manager.cc:4841] TS 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771): tablet c14125ec9fad4fbbb6104c0cdbea8d10 (table test-table [id=801931c0dd7a4971b13147e29a87361b]) successfully deleted
I20260928 23:54:42.023763 23165 log.cc:1196] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/c14125ec9fad4fbbb6104c0cdbea8d10
I20260928 23:54:42.023823 23175 tablet_bootstrap.cc:654] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:42.024039 23165 ts_tablet_manager.cc:1939] T c14125ec9fad4fbbb6104c0cdbea8d10 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting consensus metadata
I20260928 23:54:42.024358 23176 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: No bootstrap required, opened a new log
I20260928 23:54:42.024350 23177 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Bootstrap starting.
I20260928 23:54:42.024453 23176 ts_tablet_manager.cc:1397] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:42.024873 23176 raft_consensus.cc:348] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.025003 23176 raft_consensus.cc:374] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:42.025038 23176 raft_consensus.cc:729] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Initialized, Role: FOLLOWER
I20260928 23:54:42.025166 23177 tablet_bootstrap.cc:654] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:42.025199 23176 consensus_queue.cc:260] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.025501 23176 ts_tablet_manager.cc:1428] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:42.026271 22561 catalog_manager.cc:4841] TS 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2:38289): tablet c14125ec9fad4fbbb6104c0cdbea8d10 (table test-table [id=801931c0dd7a4971b13147e29a87361b]) successfully deleted
I20260928 23:54:42.026554 23177 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: No bootstrap required, opened a new log
I20260928 23:54:42.026654 23177 ts_tablet_manager.cc:1397] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:54:42.027083 23177 raft_consensus.cc:348] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.027210 23177 raft_consensus.cc:374] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:42.027245 23177 raft_consensus.cc:729] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635a344b9bfd42dea0edc1c1215a2c5a, State: Initialized, Role: FOLLOWER
I20260928 23:54:42.027372 23177 consensus_queue.cc:260] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.027639 23177 ts_tablet_manager.cc:1428] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:54:42.029171 23175 tablet_bootstrap.cc:492] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8: No bootstrap required, opened a new log
I20260928 23:54:42.029279 23175 ts_tablet_manager.cc:1397] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 23:54:42.029736 23175 raft_consensus.cc:348] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.029837 23175 raft_consensus.cc:374] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:42.029860 23175 raft_consensus.cc:729] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155ffdae490948b9831d9fea14bf18f8, State: Initialized, Role: FOLLOWER
I20260928 23:54:42.029986 23175 consensus_queue.cc:260] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.030299 23175 ts_tablet_manager.cc:1428] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:54:42.048542 23147 raft_consensus.cc:482] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:42.048683 23147 raft_consensus.cc:504] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.049171 23147 leader_election.cc:288] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:42.049654 22733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de28d9c5c1324b56a4672b7ecbee5297" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8" is_pre_election: true
I20260928 23:54:42.049746 23013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de28d9c5c1324b56a4672b7ecbee5297" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" is_pre_election: true
I20260928 23:54:42.049866 22733 raft_consensus.cc:2454] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:42.049906 23013 raft_consensus.cc:2454] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 0.
I20260928 23:54:42.050217 22806 leader_election.cc:302] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155ffdae490948b9831d9fea14bf18f8, 4dd1a8638a4b413ba059bc80bf2bc75f; no voters:
I20260928 23:54:42.050452 23147 raft_consensus.cc:2781] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:42.050518 23147 raft_consensus.cc:482] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:42.050565 23147 raft_consensus.cc:3037] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:42.051613 23147 raft_consensus.cc:504] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.051970 23147 leader_election.cc:288] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Requested vote from peers 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591), 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771)
I20260928 23:54:42.052409 22733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de28d9c5c1324b56a4672b7ecbee5297" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155ffdae490948b9831d9fea14bf18f8"
I20260928 23:54:42.052470 23013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de28d9c5c1324b56a4672b7ecbee5297" candidate_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635a344b9bfd42dea0edc1c1215a2c5a"
I20260928 23:54:42.052537 22733 raft_consensus.cc:3037] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:42.052634 23013 raft_consensus.cc:3037] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:42.053571 22733 raft_consensus.cc:2454] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:42.053591 23013 raft_consensus.cc:2454] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd1a8638a4b413ba059bc80bf2bc75f in term 1.
I20260928 23:54:42.053936 22806 leader_election.cc:302] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd1a8638a4b413ba059bc80bf2bc75f, 635a344b9bfd42dea0edc1c1215a2c5a; no voters:
I20260928 23:54:42.054164 23147 raft_consensus.cc:2781] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:42.054309 23147 raft_consensus.cc:686] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 LEADER]: Becoming Leader. State: Replica: 4dd1a8638a4b413ba059bc80bf2bc75f, State: Running, Role: LEADER
I20260928 23:54:42.054461 23147 consensus_queue.cc:237] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } }
I20260928 23:54:42.056434 22577 catalog_manager.cc:5495] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd1a8638a4b413ba059bc80bf2bc75f (127.20.88.2). New cstate: current_term: 1 leader_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dd1a8638a4b413ba059bc80bf2bc75f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 } health_report { overall_health: UNKNOWN } } }
I20260928 23:54:42.079875 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:42.076874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.20.88.2:38289)
0928 23:54:42.077149 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:42.077155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:42.077167 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:42.077205 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:42.077207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:42.077246 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:42.077316 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:42.077544 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:42.078069 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:42.078827 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:42.079132 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:42.079208 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:42.079428 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:42.079498 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:42.079550 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:42.079636 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:54:42.079680 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":121,"threads_started":1}
I20260928 23:54:42.085863 23013 raft_consensus.cc:1261] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:42.085945 22733 raft_consensus.cc:1261] T de28d9c5c1324b56a4672b7ecbee5297 P 155ffdae490948b9831d9fea14bf18f8 [term 1 FOLLOWER]: Refusing update from remote peer 4dd1a8638a4b413ba059bc80bf2bc75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:42.086288 23147 consensus_queue.cc:1035] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "635a344b9bfd42dea0edc1c1215a2c5a" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:42.086490 23106 consensus_queue.cc:1035] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "155ffdae490948b9831d9fea14bf18f8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:54:42.100628 23145 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:54:43.659914 22577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:43.660070 22577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:43.663169 22577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b22c77ed83840aca8dea7d99a5d3e4b: Sending DeleteTablet for 3 replicas of tablet de28d9c5c1324b56a4672b7ecbee5297
I20260928 23:54:43.663940 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:43 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:53416
I20260928 23:54:43.664134 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:43 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:35262
I20260928 23:54:43.664119 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet de28d9c5c1324b56a4672b7ecbee5297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:43 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48946
I20260928 23:54:43.664422 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22649
I20260928 23:54:43.665237 23222 tablet_replica.cc:330] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: stopping tablet replica
I20260928 23:54:43.665408 23222 raft_consensus.cc:2229] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:43.665544 23222 raft_consensus.cc:2258] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:43.666249 23222 ts_tablet_manager.cc:1905] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:43.667114 23220 tablet_replica.cc:330] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: stopping tablet replica
I20260928 23:54:43.667290 23220 raft_consensus.cc:2229] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:43.667534 23220 raft_consensus.cc:2258] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:43.668308 23220 ts_tablet_manager.cc:1905] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:43.668910 23222 ts_tablet_manager.cc:1918] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:54:43.669013 23222 log.cc:1196] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/de28d9c5c1324b56a4672b7ecbee5297
I20260928 23:54:43.669287 23222 ts_tablet_manager.cc:1939] T de28d9c5c1324b56a4672b7ecbee5297 P 635a344b9bfd42dea0edc1c1215a2c5a: Deleting consensus metadata
I20260928 23:54:43.670186 22562 catalog_manager.cc:4841] TS 635a344b9bfd42dea0edc1c1215a2c5a (127.20.88.3:43771): tablet de28d9c5c1324b56a4672b7ecbee5297 (table test-table [id=7076f4b6cb264b3b9bacc189ace80414]) successfully deleted
W20260928 23:54:43.671911 22562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:46591 (error 108)
I20260928 23:54:43.671984 23220 ts_tablet_manager.cc:1918] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:54:43.672099 23220 log.cc:1196] T de28d9c5c1324b56a4672b7ecbee5297 P 4dd1a8638a4b413ba059bc80bf2bc75f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/de28d9c5c1324b56a4672b7ecbee5297
I20260928 23:54:43.672101 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22789
I20260928 23:54:43.674007 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:43.673514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:46591 (local address 127.0.0.1:35270)
0928 23:54:43.673794 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:43.673910 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
W20260928 23:54:43.674197 22562 catalog_manager.cc:4584] TS 155ffdae490948b9831d9fea14bf18f8 (127.20.88.1:46591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de28d9c5c1324b56a4672b7ecbee5297: Network error: Client connection negotiation failed: client connection to 127.20.88.1:46591: connect: Connection refused (error 111)
I20260928 23:54:43.680323 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22929
I20260928 23:54:43.681334 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:43.680635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38289 (local address 127.0.0.1:53434)
0928 23:54:43.681184 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:43.681250 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:38289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
I20260928 23:54:43.687382 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 22545
2026-09-28T23:54:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6547 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:43 dist-test-slave-wvpw krb5kdc[23227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:43 dist-test-slave-wvpw krb5kdc[23227](info): set up 2 sockets
Sep 28 23:54:43 dist-test-slave-wvpw krb5kdc[23227](info): commencing operation
krb5kdc: starting...
W20260928 23:54:44.744812 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:44 dist-test-slave-wvpw krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:44.875765 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37415
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:44.992071 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:44.992450 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:44.992499 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:44.996673 23240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:44.996760 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:44.996781 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:44.996799 23240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:44.996815 23240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:45.003134 23240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:45.004377 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:45.005458 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:45.015352 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.015345 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.015345 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:45.015369 23240 server_base.cc:1034] running on GCE node
I20260928 23:54:45.016352 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:45.017598 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:45.018818 23240 hybrid_clock.cc:648] HybridClock initialized: now 1790639685018779 us; error 34 us; skew 500 ppm
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:45.022042 23240 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:45.023155 23240 webserver.cc:460] Webserver started at http://127.20.88.62:42337/ using document root <none> and password file <none>
I20260928 23:54:45.023800 23240 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:45.023869 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:45.024137 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:45.026496 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "96ce7d6826f04453a5165c6bf9668fc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "fb6356963d0d83e16c4fb362901b5c8b"
server_key_iv: "a70a4019c15946bb751ac8b7aec1e03b"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.027125 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "96ce7d6826f04453a5165c6bf9668fc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "fb6356963d0d83e16c4fb362901b5c8b"
server_key_iv: "a70a4019c15946bb751ac8b7aec1e03b"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.031387 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:54:45.034349 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.036075 23240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:45.036285 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "96ce7d6826f04453a5165c6bf9668fc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "fb6356963d0d83e16c4fb362901b5c8b"
server_key_iv: "a70a4019c15946bb751ac8b7aec1e03b"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.036453 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:45.049608 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:45.053416 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:45.053647 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:45.061982 23240 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:37415
I20260928 23:54:45.062004 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:37415 every 8 connection(s)
I20260928 23:54:45.063076 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:45.066223 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:45.072781 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9: Bootstrap starting.
I20260928 23:54:45.072893 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23240
I20260928 23:54:45.072992 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:45.073303 20832 external_mini_cluster.cc:1428] Setting key d1497cbc1727a9cb46659948ba3176a1
I20260928 23:54:45.075845 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:45.076773 23308 log.cc:824] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:45.079098 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9: No bootstrap required, opened a new log
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:45.084074 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } }
I20260928 23:54:45.084436 23308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:45.084548 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ce7d6826f04453a5165c6bf9668fc9, State: Initialized, Role: FOLLOWER
I20260928 23:54:45.085079 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } }
I20260928 23:54:45.085239 23308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:45.085316 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:45.085419 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:45.086607 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } }
I20260928 23:54:45.087060 23308 leader_election.cc:302] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96ce7d6826f04453a5165c6bf9668fc9; no voters:
I20260928 23:54:45.087484 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:45.087600 23313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:45.087841 23313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 96ce7d6826f04453a5165c6bf9668fc9, State: Running, Role: LEADER
I20260928 23:54:45.088326 23313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } }
I20260928 23:54:45.088943 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:45.089742 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96ce7d6826f04453a5165c6bf9668fc9. Latest consensus state: current_term: 1 leader_uuid: "96ce7d6826f04453a5165c6bf9668fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } } }
I20260928 23:54:45.089973 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:45.090721 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96ce7d6826f04453a5165c6bf9668fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ce7d6826f04453a5165c6bf9668fc9" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37415 } } }
I20260928 23:54:45.090828 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:45.091017 23322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:45.092381 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.074883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.20.88.62:37415)
0928 23:54:45.075367 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:45.075384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:45.075423 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:54:45.076208 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:45.076299 (+ 91us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:45.076689 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:45.077090 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:45.077996 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.079075 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.080048 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.080815 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.080990 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.083731 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:45.083738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:45.083758 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:45.083794 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:45.086027 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.086611 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.086617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.086623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.086842 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.087206 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.087210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.087213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.087840 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:45.088852 (+ 1012us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:45.089017 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 23:54:45.089348 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":150,"threads_started":1}
I20260928 23:54:45.095613 23322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:45.101898 23322 catalog_manager.cc:1347] Generated new cluster ID: 8a2f2517857b4d77a3c5ca889c74e4a5
I20260928 23:54:45.101999 23322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:45.114794 23322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:45.115829 23322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:45.123867 23322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96ce7d6826f04453a5165c6bf9668fc9: Generated new TSK 0
I20260928 23:54:45.124760 23322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:45.159734 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:45.281481 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:45.281785 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:45.281831 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:45.286237 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:45.286316 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:45.286379 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:45.292573 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:45.293780 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:45.294895 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:45.305238 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:45.305397 23334 server_base.cc:1034] running on GCE node
W20260928 23:54:45.305428 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.305712 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:45.306186 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:45.306922 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:45.308113 23334 hybrid_clock.cc:648] HybridClock initialized: now 1790639685308086 us; error 31 us; skew 500 ppm
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:45.311445 23334 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:45.312453 23334 webserver.cc:460] Webserver started at http://127.20.88.1:44083/ using document root <none> and password file <none>
I20260928 23:54:45.313027 23334 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:45.313092 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:45.313269 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:45.315460 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "05ecd0f17d764f5f94aa742bca7a4dc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "252cd25227c33601b315f0e98b90b259"
server_key_iv: "5416d6302a4914eddb76076dda48fcf9"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.316138 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "05ecd0f17d764f5f94aa742bca7a4dc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "252cd25227c33601b315f0e98b90b259"
server_key_iv: "5416d6302a4914eddb76076dda48fcf9"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.320546 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:54:45.323664 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.325387 23334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:45.325515 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "05ecd0f17d764f5f94aa742bca7a4dc9"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "252cd25227c33601b315f0e98b90b259"
server_key_iv: "5416d6302a4914eddb76076dda48fcf9"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.325635 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:45.358147 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:45.361795 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:45.361969 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:45.362550 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:45.363397 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:45.363446 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.363490 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:45.363505 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.373416 23334 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35967
I20260928 23:54:45.373484 23462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35967 every 8 connection(s)
I20260928 23:54:45.374471 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:45.377254 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23334
I20260928 23:54:45.377417 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:45.377720 20832 external_mini_cluster.cc:1428] Setting key 0f06f8780de91c2b993fdac3a1ba9873
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:54:45.391139 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.376848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:52751 (local address 127.20.88.62:37415)
0928 23:54:45.377039 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:45.377044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:45.378346 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 23:54:45.379346 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:45.379349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:45.379401 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:45.379481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:45.380821 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.381379 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.383306 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.383549 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.383644 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.386931 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:45.386935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:45.386937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:45.386964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:45.388736 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.389369 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.389391 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.389393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.389490 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.389857 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.389861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.389863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.390150 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:45.390275 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:45.390828 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 23:54:45.390965 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:45.392408 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.377338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37415 (local address 127.20.88.1:52751)
0928 23:54:45.378177 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:45.378238 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:45.379127 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:45.379680 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:45.379688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:45.380067 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:45.380629 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.380643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.381524 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.381531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.383130 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.383146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.383787 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.383794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.384049 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.384837 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:45.384862 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:45.386718 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:45.388882 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.388889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.388908 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.389249 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.389590 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.389594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.389597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.389758 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.390267 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:45.390275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:45.390568 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 23:54:45.390781 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 23:54:45.391231 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":144,"threads_started":1}
I20260928 23:54:45.394080 23463 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37415
I20260928 23:54:45.394399 23463 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:45.395038 23463 heartbeater.cc:511] Master 127.20.88.62:37415 requested a full tablet report, sending...
I20260928 23:54:45.397074 23272 ts_manager.cc:194] Registered new tserver with Master: 05ecd0f17d764f5f94aa742bca7a4dc9 (127.20.88.1:35967)
I20260928 23:54:45.399101 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:52751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:45.411685 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:45.529194 23468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:45.529541 23468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:45.529620 23468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:45.534034 23468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:45.534140 23468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:45.534241 23468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:45.539979 23468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:45.541342 23468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:45.542532 23468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:45.554477 23476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.554477 23473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.554497 23474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:45.554885 23468 server_base.cc:1034] running on GCE node
I20260928 23:54:45.555425 23468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:45.556377 23468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:45.557595 23468 hybrid_clock.cc:648] HybridClock initialized: now 1790639685557559 us; error 49 us; skew 500 ppm
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:45.561085 23468 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:45.562198 23468 webserver.cc:460] Webserver started at http://127.20.88.2:39777/ using document root <none> and password file <none>
I20260928 23:54:45.562849 23468 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:45.562947 23468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:45.563185 23468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:45.565503 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "78b84c57b5f84367b06ef26898b68959"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "b2547b76cecd7c5fe41299beb34269ac"
server_key_iv: "66e67f40b46742f8250ab722f142581f"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.566157 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "78b84c57b5f84367b06ef26898b68959"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "b2547b76cecd7c5fe41299beb34269ac"
server_key_iv: "66e67f40b46742f8250ab722f142581f"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.570412 23468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:45.573369 23483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.574816 23468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:45.574985 23468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "78b84c57b5f84367b06ef26898b68959"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "b2547b76cecd7c5fe41299beb34269ac"
server_key_iv: "66e67f40b46742f8250ab722f142581f"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.575124 23468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:45.594396 23468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:45.598124 23468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:45.598364 23468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:45.599028 23468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:45.599931 23468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:45.600008 23468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.600081 23468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:45.600140 23468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.610013 23468 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:36451
I20260928 23:54:45.610033 23596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:36451 every 8 connection(s)
I20260928 23:54:45.611040 23468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:45.619426 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23468
I20260928 23:54:45.619571 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:45.619879 20832 external_mini_cluster.cc:1428] Setting key 987e515ce4e75675ce38b39499684386
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:45.626993 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.613161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:58317 (local address 127.20.88.62:37415)
0928 23:54:45.613319 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:45.613323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:45.614155 (+ 832us) server_negotiation.cc:407] Connection header received
0928 23:54:45.615505 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:45.615508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:45.615560 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:45.615642 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:45.617104 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.617725 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.619101 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.619339 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.619427 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.622602 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:45.622605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:45.622608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:45.622642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:45.624377 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.625071 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.625073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.625075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.625168 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.625637 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.625640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.625642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.625855 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:45.625940 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:45.626732 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 23:54:45.626861 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:54:45.628502 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.613512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37415 (local address 127.20.88.2:58317)
0928 23:54:45.613975 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:45.614023 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:45.615275 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:45.615865 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:45.615873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:45.616372 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:45.616934 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.616949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.617885 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.617888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.618951 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.618962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.619534 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.619540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.619852 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.620700 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:45.620722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:45.622413 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:45.624506 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.624516 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.624540 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.624955 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.625265 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.625272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.625277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.625523 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.625982 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:45.625992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:45.626419 (+ 427us) client_negotiation.cc:769] Sending connection context
0928 23:54:45.626678 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 23:54:45.627121 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
I20260928 23:54:45.629943 23597 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37415
I20260928 23:54:45.630261 23597 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:54:45.630852 23597 heartbeater.cc:511] Master 127.20.88.62:37415 requested a full tablet report, sending...
I20260928 23:54:45.631978 23272 ts_manager.cc:194] Registered new tserver with Master: 78b84c57b5f84367b06ef26898b68959 (127.20.88.2:36451)
I20260928 23:54:45.632659 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:58317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:45.651836 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:45.767215 23602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:45.767586 23602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:45.767679 23602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:45.771915 23602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:45.772053 23602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:45.772157 23602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:45.777931 23602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:45.779273 23602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:45.780532 23602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:45.791222 23610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.791222 23607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:45.791445 23608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:45.791656 23602 server_base.cc:1034] running on GCE node
I20260928 23:54:45.792114 23602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:45.792876 23602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:45.794075 23602 hybrid_clock.cc:648] HybridClock initialized: now 1790639685794055 us; error 33 us; skew 500 ppm
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:45.797402 23602 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:45.798367 23602 webserver.cc:460] Webserver started at http://127.20.88.3:42535/ using document root <none> and password file <none>
I20260928 23:54:45.798942 23602 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:45.799003 23602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:45.799176 23602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:45.801491 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "bbc814410eed40e8bb94852a7b7247eb"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "8346751a58f16f1a495345d82e807fdf"
server_key_iv: "081e0a7fc8a6049a8a9d70f5cdade4fb"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.802093 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "bbc814410eed40e8bb94852a7b7247eb"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "8346751a58f16f1a495345d82e807fdf"
server_key_iv: "081e0a7fc8a6049a8a9d70f5cdade4fb"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.806326 23602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:45.809144 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.810540 23602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:45.810698 23602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "bbc814410eed40e8bb94852a7b7247eb"
format_stamp: "Formatted at 2026-09-28 23:54:45 on dist-test-slave-wvpw"
server_key: "8346751a58f16f1a495345d82e807fdf"
server_key_iv: "081e0a7fc8a6049a8a9d70f5cdade4fb"
server_key_version: "encryptionkey@0"
I20260928 23:54:45.810827 23602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:45.822198 23602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:45.825800 23602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:45.826025 23602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:45.826653 23602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:45.827520 23602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:45.827593 23602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.827668 23602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:45.827713 23602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:45.837292 23602 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:36753
I20260928 23:54:45.837309 23730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:36753 every 8 connection(s)
I20260928 23:54:45.838310 23602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:45.838908 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23602
I20260928 23:54:45.839028 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:45.839341 20832 external_mini_cluster.cc:1428] Setting key a96c5f3072db453063796ff204aa55f5
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639685, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:45.853686 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.840203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:42723 (local address 127.20.88.62:37415)
0928 23:54:45.840384 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:45.840390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:45.841617 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 23:54:45.843062 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:45.843065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:45.843124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:45.843204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:45.844609 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.845177 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.846458 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.846705 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.846829 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.849663 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:45.849666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:45.849668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:45.849698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:45.851009 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.851592 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.851596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.851598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.851689 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:45.852081 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:45.852084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:45.852086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:45.852374 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:45.852497 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:45.853353 (+ 856us) server_negotiation.cc:299] Negotiation successful
0928 23:54:45.853524 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 23:54:45.854748 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:45.840901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37415 (local address 127.20.88.3:42723)
0928 23:54:45.841446 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:45.841512 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:45.842833 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:45.843396 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:45.843405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:45.843810 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:45.844438 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.844456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.845358 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.845362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.846317 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:45.846327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:45.846884 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:45.846888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:45.847083 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:45.847731 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:45.847749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:45.849449 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:45.851143 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.851150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.851169 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.851473 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.851802 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:45.851805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:45.851807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:45.851974 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:45.852521 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:45.852528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:45.852975 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 23:54:45.853234 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 23:54:45.853652 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":111,"threads_started":1}
I20260928 23:54:45.855998 23731 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37415
I20260928 23:54:45.856300 23731 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:45.856854 23731 heartbeater.cc:511] Master 127.20.88.62:37415 requested a full tablet report, sending...
I20260928 23:54:45.858088 23272 ts_manager.cc:194] Registered new tserver with Master: bbc814410eed40e8bb94852a7b7247eb (127.20.88.3:36753)
I20260928 23:54:45.859058 23272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:42723
I20260928 23:54:45.866261 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:54:45.880656 23311 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:45.878821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.20.88.62:37415)
0928 23:54:45.878992 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:45.878998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:45.879195 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:54:45.879324 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:45.879326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:45.879375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:45.879459 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:45.880484 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53888: BlockingRecv error: recv got EOF from 127.0.0.1:53888 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:54:45.882282 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23334
I20260928 23:54:45.887881 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23468
I20260928 23:54:45.893858 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23602
I20260928 23:54:45.899642 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23240
2026-09-28T23:54:45Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2210 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23743](info): set up 2 sockets
Sep 28 23:54:45 dist-test-slave-wvpw krb5kdc[23743](info): commencing operation
krb5kdc: starting...
W20260928 23:54:46.940941 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:47.071722 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35723
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:47.184594 23756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:47.184940 23756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:47.185024 23756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:47.189239 23756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:47.189339 23756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:47.189368 23756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:47.189397 23756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:47.189436 23756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:47.195084 23756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:47.196381 23756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:47.197490 23756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:47.207013 23764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:47.207024 23762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:47.207058 23756 server_base.cc:1034] running on GCE node
W20260928 23:54:47.207010 23761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:47.207990 23756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:47.209336 23756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:47.210522 23756 hybrid_clock.cc:648] HybridClock initialized: now 1790639687210502 us; error 39 us; skew 500 ppm
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639687, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:47.223076 23756 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:47.224152 23756 webserver.cc:460] Webserver started at http://127.20.88.62:43543/ using document root <none> and password file <none>
I20260928 23:54:47.224833 23756 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:47.224900 23756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:47.225078 23756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:47.227272 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "a0bdcc37270c4c8db480479d4b6e5125"
format_stamp: "Formatted at 2026-09-28 23:54:47 on dist-test-slave-wvpw"
server_key: "ef6b733cb4e2f5da1e529e1ac4fc0fc4"
server_key_iv: "005b839159abcf700a201fe9420800ff"
server_key_version: "encryptionkey@0"
I20260928 23:54:47.227842 23756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "a0bdcc37270c4c8db480479d4b6e5125"
format_stamp: "Formatted at 2026-09-28 23:54:47 on dist-test-slave-wvpw"
server_key: "ef6b733cb4e2f5da1e529e1ac4fc0fc4"
server_key_iv: "005b839159abcf700a201fe9420800ff"
server_key_version: "encryptionkey@0"
I20260928 23:54:47.231940 23756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:47.234767 23771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:47.236307 23756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:54:47.236465 23756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "a0bdcc37270c4c8db480479d4b6e5125"
format_stamp: "Formatted at 2026-09-28 23:54:47 on dist-test-slave-wvpw"
server_key: "ef6b733cb4e2f5da1e529e1ac4fc0fc4"
server_key_iv: "005b839159abcf700a201fe9420800ff"
server_key_version: "encryptionkey@0"
I20260928 23:54:47.236579 23756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:47.256014 23756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:47.260082 23756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:47.260365 23756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:47.268078 23756 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35723
I20260928 23:54:47.268780 23823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35723 every 8 connection(s)
I20260928 23:54:47.269302 23756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:47.273188 23824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:47.279419 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23756
I20260928 23:54:47.279526 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:47.279731 23824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125: Bootstrap starting.
I20260928 23:54:47.279829 20832 external_mini_cluster.cc:1428] Setting key c54159169ec8dff03478b430eed625ee
I20260928 23:54:47.282826 23824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:47.283751 23824 log.cc:824] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:47.286077 23824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125: No bootstrap required, opened a new log
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:47.291038 23824 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } }
I20260928 23:54:47.291419 23824 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:47.291498 23824 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0bdcc37270c4c8db480479d4b6e5125, State: Initialized, Role: FOLLOWER
I20260928 23:54:47.292122 23824 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } }
I20260928 23:54:47.292384 23824 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:47.292466 23824 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:47.292588 23824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:47.293648 23824 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } }
I20260928 23:54:47.294056 23824 leader_election.cc:302] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0bdcc37270c4c8db480479d4b6e5125; no voters:
I20260928 23:54:47.294492 23824 leader_election.cc:288] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:47.294584 23829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:47.294798 23829 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [term 1 LEADER]: Becoming Leader. State: Replica: a0bdcc37270c4c8db480479d4b6e5125, State: Running, Role: LEADER
I20260928 23:54:47.295148 23829 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } }
I20260928 23:54:47.296038 23824 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:47.296768 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0bdcc37270c4c8db480479d4b6e5125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } } }
I20260928 23:54:47.296902 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:47.297199 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0bdcc37270c4c8db480479d4b6e5125. Latest consensus state: current_term: 1 leader_uuid: "a0bdcc37270c4c8db480479d4b6e5125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0bdcc37270c4c8db480479d4b6e5125" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35723 } } }
I20260928 23:54:47.297308 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:47.297418 23833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:47.298199 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:47.281693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.20.88.62:35723)
0928 23:54:47.282158 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:47.282184 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:47.282223 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:54:47.282920 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:47.282940 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:47.283367 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:47.283677 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:47.284933 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:47.286015 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:47.287215 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:47.287745 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:47.287882 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:47.291072 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:47.291081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:47.291103 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:47.291144 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:47.293726 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:47.294324 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:47.294334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:47.294345 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:47.294523 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:47.294901 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:47.294904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:47.294906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:47.295595 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:47.295747 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:47.296343 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 23:54:47.296720 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":166,"threads_started":1}
I20260928 23:54:47.301261 23833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:47.302919 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:47.307968 23833 catalog_manager.cc:1347] Generated new cluster ID: 471acaa4bcc4466bae481be7125eaac8
I20260928 23:54:47.308053 23833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:47.324566 23827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:47.318243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.20.88.62:35723)
0928 23:54:47.318384 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:47.318389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:47.318782 (+ 393us) server_negotiation.cc:407] Connection header received
0928 23:54:47.318912 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:47.318916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:47.318974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:47.319072 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:47.319708 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:47.320569 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:47.321874 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:47.322274 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:47.322389 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:47.323444 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:47.323448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:47.323451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:47.324010 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:47.324045 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:47.324065 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:47.324202 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:47.324327 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:54:47.324404 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:54:47.325371 23833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:47.326683 23833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:47.337137 23833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0bdcc37270c4c8db480479d4b6e5125: Generated new TSK 0
I20260928 23:54:47.338068 23833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:47.344801 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23756
2026-09-28T23:54:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1442 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23857](info): set up 2 sockets
Sep 28 23:54:47 dist-test-slave-wvpw krb5kdc[23857](info): commencing operation
krb5kdc: starting...
W20260928 23:54:48.397015 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:48 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:48.525413 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42593
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:48.641887 23870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:48.642197 23870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:48.642241 23870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:48.646516 23870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:48.646596 23870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:48.646616 23870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:48.646633 23870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:48.646649 23870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:48.653002 23870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:48.654175 23870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:48.655261 23870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:48.664433 23875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:48.664433 23878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:48.664470 23876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:48.664881 23870 server_base.cc:1034] running on GCE node
I20260928 23:54:48.665723 23870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:48.666962 23870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:48.668164 23870 hybrid_clock.cc:648] HybridClock initialized: now 1790639688668151 us; error 31 us; skew 500 ppm
Sep 28 23:54:48 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:48.671284 23870 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:48.672431 23870 webserver.cc:460] Webserver started at http://127.20.88.62:37021/ using document root <none> and password file <none>
I20260928 23:54:48.673043 23870 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:48.673143 23870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:48.673393 23870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:48.675626 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "c5def9cf41a74092a55586637435b8a4"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "28ae147f4b6f678b05d135e9fbb9e7ce"
server_key_iv: "aced6c73dcfc390746fe562f3eea8a32"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.676314 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "c5def9cf41a74092a55586637435b8a4"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "28ae147f4b6f678b05d135e9fbb9e7ce"
server_key_iv: "aced6c73dcfc390746fe562f3eea8a32"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.680408 23870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:54:48.683148 23885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:48.684551 23870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:48.684737 23870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "c5def9cf41a74092a55586637435b8a4"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "28ae147f4b6f678b05d135e9fbb9e7ce"
server_key_iv: "aced6c73dcfc390746fe562f3eea8a32"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.684876 23870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:48.699056 23870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:48.702656 23870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:48.702895 23870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:48.710955 23870 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42593
I20260928 23:54:48.711265 23937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42593 every 8 connection(s)
I20260928 23:54:48.712162 23870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:48.712433 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23870
I20260928 23:54:48.712548 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:48.712895 20832 external_mini_cluster.cc:1428] Setting key 02843e5561454da12ffb1fc3d193cde4
I20260928 23:54:48.715922 23938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:48.722222 23938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: Bootstrap starting.
Sep 28 23:54:48 dist-test-slave-wvpw krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:48.725584 23938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:48.726395 23938 log.cc:824] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:48.728920 23938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: No bootstrap required, opened a new log
I20260928 23:54:48.729310 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:48.714410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.20.88.62:42593)
0928 23:54:48.715348 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:48.715364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:48.715402 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:54:48.716126 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:48.716149 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:48.716586 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:48.716955 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:48.717789 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:48.718916 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:48.719810 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:48.720500 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:48.720672 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:48.723748 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:48.723754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:48.723770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:48.723807 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:48.725890 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:48.726405 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:48.726412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:48.726426 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:48.726596 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:48.726926 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:48.726929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:48.726930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:48.727439 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:48.727632 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:48.727939 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:54:48.728285 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":152,"threads_started":1}
I20260928 23:54:48.732620 23938 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } }
I20260928 23:54:48.732870 23938 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:48.732906 23938 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5def9cf41a74092a55586637435b8a4, State: Initialized, Role: FOLLOWER
I20260928 23:54:48.733291 23938 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } }
I20260928 23:54:48.733405 23938 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:48.733444 23938 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:48.733515 23938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:48.734473 23938 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } }
I20260928 23:54:48.734787 23938 leader_election.cc:302] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5def9cf41a74092a55586637435b8a4; no voters:
I20260928 23:54:48.735065 23938 leader_election.cc:288] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:48.735210 23943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:48.735481 23943 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [term 1 LEADER]: Becoming Leader. State: Replica: c5def9cf41a74092a55586637435b8a4, State: Running, Role: LEADER
I20260928 23:54:48.735929 23943 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } }
I20260928 23:54:48.736586 23938 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:48.737983 23945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5def9cf41a74092a55586637435b8a4. Latest consensus state: current_term: 1 leader_uuid: "c5def9cf41a74092a55586637435b8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } } }
I20260928 23:54:48.737998 23944 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5def9cf41a74092a55586637435b8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5def9cf41a74092a55586637435b8a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42593 } } }
I20260928 23:54:48.738155 23945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:48.738154 23944 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4 [sys.catalog]: This master's current role is: LEADER
W20260928 23:54:48.741945 23958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:48.742036 23958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:48.742138 23954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:48.742902 23954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:48.749466 23954 catalog_manager.cc:1347] Generated new cluster ID: cbe5fb68e14b488aa489e4d138ea78dd
I20260928 23:54:48.749588 23954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:48.756114 23954 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:48.757441 23954 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:48.778757 23954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5def9cf41a74092a55586637435b8a4: Generated new TSK 0
I20260928 23:54:48.779740 23954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:48.819864 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:48.937793 23964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:48.938185 23964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:48.938282 23964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:48.942672 23964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:48.942787 23964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:48.942884 23964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:48.948570 23964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:48.949836 23964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:48.950925 23964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:48.961046 23970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:48.961027 23969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:48.961076 23972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:48.961364 23964 server_base.cc:1034] running on GCE node
I20260928 23:54:48.961834 23964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:48.962598 23964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:48.963797 23964 hybrid_clock.cc:648] HybridClock initialized: now 1790639688963778 us; error 30 us; skew 500 ppm
Sep 28 23:54:48 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:48.966771 23964 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:48.967708 23964 webserver.cc:460] Webserver started at http://127.20.88.1:33641/ using document root <none> and password file <none>
I20260928 23:54:48.968369 23964 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:48.968438 23964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:48.968680 23964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:48.970777 23964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "e4b40a94822b4442976c351f34fe8e0c"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "7805ec56138abcd7c83841d667a2bb8c"
server_key_iv: "24f779a2c883397dc10c83a02434ea88"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.971354 23964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "e4b40a94822b4442976c351f34fe8e0c"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "7805ec56138abcd7c83841d667a2bb8c"
server_key_iv: "24f779a2c883397dc10c83a02434ea88"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.975346 23964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:48.977988 23979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:48.979364 23964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:54:48.979470 23964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "e4b40a94822b4442976c351f34fe8e0c"
format_stamp: "Formatted at 2026-09-28 23:54:48 on dist-test-slave-wvpw"
server_key: "7805ec56138abcd7c83841d667a2bb8c"
server_key_iv: "24f779a2c883397dc10c83a02434ea88"
server_key_version: "encryptionkey@0"
I20260928 23:54:48.979615 23964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:48.999712 23964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:49.003904 23964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:49.004144 23964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:49.004837 23964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:49.005733 23964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:49.005806 23964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.005877 23964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:49.005908 23964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.015036 23964 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:46585
I20260928 23:54:49.015079 24092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:46585 every 8 connection(s)
I20260928 23:54:49.016021 23964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:49.017521 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 23964
I20260928 23:54:49.017665 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:49.018033 20832 external_mini_cluster.cc:1428] Setting key 522fc67c39a096fde2126bfc4d8891a6
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:49.032974 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.018206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57993 (local address 127.20.88.62:42593)
0928 23:54:49.018391 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:49.018396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:49.019422 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 23:54:49.021079 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:49.021086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:49.021158 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:49.021217 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:49.023266 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.023911 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.025392 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.025641 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.025767 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.028520 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:49.028524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:49.028526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:49.028556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:49.030143 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.030962 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.030986 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.030991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.031111 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.031515 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.031521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.031526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.031822 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:49.031937 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:49.032615 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 23:54:49.032784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:54:49.033875 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.018749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42593 (local address 127.20.88.1:57993)
0928 23:54:49.019266 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:49.019325 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:49.020800 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:49.021723 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:49.021736 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:49.022263 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:49.023051 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.023076 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.024130 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.024136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.025249 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.025260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.025818 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.025822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.026005 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.026697 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:49.026743 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:49.028314 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:49.030375 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.030382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.030401 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.030830 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.031235 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.031237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.031239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.031395 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.031944 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:49.031949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:49.032315 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 23:54:49.032547 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 23:54:49.032920 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260928 23:54:49.035501 24093 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42593
I20260928 23:54:49.035882 24093 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:49.036592 24093 heartbeater.cc:511] Master 127.20.88.62:42593 requested a full tablet report, sending...
I20260928 23:54:49.038945 23902 ts_manager.cc:194] Registered new tserver with Master: e4b40a94822b4442976c351f34fe8e0c (127.20.88.1:46585)
I20260928 23:54:49.040760 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:57993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:49.049701 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:49.170441 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:49.170760 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:49.170809 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:49.175056 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:49.175149 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:49.175212 24098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:49.180887 24098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:49.181975 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:49.183040 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:49.193239 24106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:49.193318 24103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:49.193348 24104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:49.193734 24098 server_base.cc:1034] running on GCE node
I20260928 23:54:49.194154 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:49.194813 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:49.196033 24098 hybrid_clock.cc:648] HybridClock initialized: now 1790639689196004 us; error 36 us; skew 500 ppm
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:49.199028 24098 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:49.199970 24098 webserver.cc:460] Webserver started at http://127.20.88.2:42105/ using document root <none> and password file <none>
I20260928 23:54:49.200609 24098 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:49.200726 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:49.200956 24098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:49.203114 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "e03dcda0879b4bf6925e3e132539d47b"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "6d48b73882b03df1f6dd2d7a3c2ea9c8"
server_key_iv: "4c6a1b2deb3c182e7ea1e57290573d9a"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.203744 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "e03dcda0879b4bf6925e3e132539d47b"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "6d48b73882b03df1f6dd2d7a3c2ea9c8"
server_key_iv: "4c6a1b2deb3c182e7ea1e57290573d9a"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.207774 24098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:54:49.210685 24113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.212087 24098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:49.212294 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "e03dcda0879b4bf6925e3e132539d47b"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "6d48b73882b03df1f6dd2d7a3c2ea9c8"
server_key_iv: "4c6a1b2deb3c182e7ea1e57290573d9a"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.212425 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:49.228209 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:49.231935 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:49.232106 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:49.232764 24098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:49.233615 24098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:49.233672 24098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.233714 24098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:49.233731 24098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.243711 24098 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:34963
I20260928 23:54:49.243732 24226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:34963 every 8 connection(s)
I20260928 23:54:49.244779 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:49.247932 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24098
I20260928 23:54:49.248055 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:49.248381 20832 external_mini_cluster.cc:1428] Setting key 47629d12a89a17dbdcf70750160483e2
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:54:49.261270 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.246744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:33559 (local address 127.20.88.62:42593)
0928 23:54:49.246893 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:49.246897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:49.247886 (+ 989us) server_negotiation.cc:407] Connection header received
0928 23:54:49.249657 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:49.249660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:49.249725 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:49.249824 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:49.251078 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.251890 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.253230 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.253645 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.253787 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.256894 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:49.256898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:49.256901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:49.256937 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:49.258710 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.259304 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.259308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.259314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.259450 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.259877 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.259881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.259882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.260088 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:49.260178 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:49.260870 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 23:54:49.261031 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:54:49.262315 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.247177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42593 (local address 127.20.88.2:33559)
0928 23:54:49.247687 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:49.247751 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:49.249431 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:49.250024 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:49.250032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:49.250412 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:49.250924 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.250938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.252046 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.252049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.253097 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.253107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.253853 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.253858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.254036 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.254819 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:49.254836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:49.256672 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:49.258836 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.258842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.258858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.259147 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.259558 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.259561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.259563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.259741 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.260232 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:49.260243 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:49.260607 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 23:54:49.260782 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 23:54:49.261178 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260928 23:54:49.263746 24227 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42593
I20260928 23:54:49.264078 24227 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:49.264746 24227 heartbeater.cc:511] Master 127.20.88.62:42593 requested a full tablet report, sending...
I20260928 23:54:49.266008 23902 ts_manager.cc:194] Registered new tserver with Master: e03dcda0879b4bf6925e3e132539d47b (127.20.88.2:34963)
I20260928 23:54:49.266659 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:33559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:49.280570 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:49.402889 24232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:49.403268 24232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:49.403360 24232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:49.407965 24232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:49.408077 24232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:49.408188 24232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:49.414204 24232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:49.415531 24232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:49.416882 24232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:49.427691 24237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:49.427871 24240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:49.428071 24232 server_base.cc:1034] running on GCE node
W20260928 23:54:49.427728 24238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:49.428691 24232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:49.429454 24232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:49.430656 24232 hybrid_clock.cc:648] HybridClock initialized: now 1790639689430618 us; error 55 us; skew 500 ppm
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:49.433755 24232 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:49.434774 24232 webserver.cc:460] Webserver started at http://127.20.88.3:35201/ using document root <none> and password file <none>
I20260928 23:54:49.435417 24232 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:49.435516 24232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:49.435761 24232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:49.438302 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "2826931565ee4eebbb10b3815263fe50"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "96a813cfbab23e6f6264c277edb53533"
server_key_iv: "7382ffdd6a2cbdcd332de8ef63f73086"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.438930 24232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "2826931565ee4eebbb10b3815263fe50"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "96a813cfbab23e6f6264c277edb53533"
server_key_iv: "7382ffdd6a2cbdcd332de8ef63f73086"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.443279 24232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 23:54:49.446161 24247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.447690 24232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:49.447893 24232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "2826931565ee4eebbb10b3815263fe50"
format_stamp: "Formatted at 2026-09-28 23:54:49 on dist-test-slave-wvpw"
server_key: "96a813cfbab23e6f6264c277edb53533"
server_key_iv: "7382ffdd6a2cbdcd332de8ef63f73086"
server_key_version: "encryptionkey@0"
I20260928 23:54:49.448047 24232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:49.462450 24232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:49.466149 24232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:49.466385 24232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:49.467027 24232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:49.467921 24232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:49.467970 24232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.468046 24232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:49.468087 24232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:49.477373 24232 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:35143
I20260928 23:54:49.477437 24360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:35143 every 8 connection(s)
I20260928 23:54:49.478439 24232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:49.478977 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24232
I20260928 23:54:49.479101 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:49.479416 20832 external_mini_cluster.cc:1428] Setting key bc8239e590981445484ee85dc79f1f19
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:49.493831 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.480469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:54899 (local address 127.20.88.62:42593)
0928 23:54:49.480596 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:49.480601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:49.481694 (+ 1093us) server_negotiation.cc:407] Connection header received
0928 23:54:49.483224 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:49.483226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:49.483280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:49.483360 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:49.484827 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.485365 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.486522 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.486746 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.486828 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.489883 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:49.489887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:49.489890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:49.489918 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:49.491296 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.491803 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.491806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.491807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.491883 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.492347 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.492351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.492352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.492607 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:49.492707 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:49.493494 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 23:54:49.493630 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:54:49.494657 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.480979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42593 (local address 127.20.88.3:54899)
0928 23:54:49.481520 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:49.481586 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:49.482988 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:49.483541 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:49.483548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:49.484026 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:49.484684 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.484700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.485482 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.485484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.486411 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:49.486419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.486892 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.486895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:49.487049 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.487760 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:49.487778 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:49.489623 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:49.491413 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.491418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.491432 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.491700 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.491984 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:49.491988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:49.491990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:49.492191 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:49.492745 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:49.492754 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:49.493117 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 23:54:49.493358 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 23:54:49.493739 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260928 23:54:49.495867 24361 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42593
I20260928 23:54:49.496088 24361 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:49.496627 24361 heartbeater.cc:511] Master 127.20.88.62:42593 requested a full tablet report, sending...
I20260928 23:54:49.497718 23902 ts_manager.cc:194] Registered new tserver with Master: 2826931565ee4eebbb10b3815263fe50 (127.20.88.3:35143)
I20260928 23:54:49.498409 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:54899
I20260928 23:54:49.506265 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639689, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:49.532359 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:49.523331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.20.88.62:42593)
0928 23:54:49.523516 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:49.523521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:49.523658 (+ 137us) server_negotiation.cc:407] Connection header received
0928 23:54:49.523826 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:49.523834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:49.523878 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:49.523967 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:49.524811 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.525355 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.526236 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:49.526456 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:49.526567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:49.529062 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:49.529066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:49.529068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:49.529101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:49.530486 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.530959 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.530964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.530965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.531054 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:49.531347 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:49.531350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:49.531352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:49.531594 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:49.531766 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:49.532014 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:54:49.532158 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 23:54:49.532763 23902 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55400
I20260928 23:54:49.533671 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23964
I20260928 23:54:49.539958 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24098
I20260928 23:54:49.545504 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24232
I20260928 23:54:49.551050 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 23870
2026-09-28T23:54:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2209 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[24373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[24373](info): set up 2 sockets
Sep 28 23:54:49 dist-test-slave-wvpw krb5kdc[24373](info): commencing operation
krb5kdc: starting...
W20260928 23:54:50.612969 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/test-user.keytab.
W20260928 23:54:51.280519 20832 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.569s user 0.000s sys 0.002s
Sep 28 23:54:51 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
W20260928 23:54:52.654663 20832 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.88.62: real 1.348s user 0.003s sys 0.000s
I20260928 23:54:52.655175 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43331
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:52.780354 24387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:52.780704 24387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:52.780751 24387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:52.785140 24387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:52.785225 24387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:52.785246 24387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:52.785264 24387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:52.785280 24387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:52.791368 24387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:52.792742 24387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:52.793896 24387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:52.803871 24392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:52.803931 24395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:52.804311 24393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:52.804388 24387 server_base.cc:1034] running on GCE node
I20260928 23:54:52.805074 24387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:52.806425 24387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:52.807657 24387 hybrid_clock.cc:648] HybridClock initialized: now 1790639692807580 us; error 56 us; skew 500 ppm
Sep 28 23:54:52 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639692, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:52.815615 24387 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:52.816862 24387 webserver.cc:460] Webserver started at http://127.20.88.62:38441/ using document root <none> and password file <none>
I20260928 23:54:52.817633 24387 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:52.817741 24387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:52.818009 24387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:52.820475 24387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "03076b413b164688b0d420d25be78c3f"
format_stamp: "Formatted at 2026-09-28 23:54:52 on dist-test-slave-wvpw"
server_key: "b3645d4fe732ca50be82dbbd0ed51603"
server_key_iv: "ccfbc3840fd33585ca64c0e9b0412849"
server_key_version: "encryptionkey@0"
I20260928 23:54:52.821144 24387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "03076b413b164688b0d420d25be78c3f"
format_stamp: "Formatted at 2026-09-28 23:54:52 on dist-test-slave-wvpw"
server_key: "b3645d4fe732ca50be82dbbd0ed51603"
server_key_iv: "ccfbc3840fd33585ca64c0e9b0412849"
server_key_version: "encryptionkey@0"
I20260928 23:54:52.825719 24387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 23:54:52.828711 24402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:52.830430 24387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:52.830610 24387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "03076b413b164688b0d420d25be78c3f"
format_stamp: "Formatted at 2026-09-28 23:54:52 on dist-test-slave-wvpw"
server_key: "b3645d4fe732ca50be82dbbd0ed51603"
server_key_iv: "ccfbc3840fd33585ca64c0e9b0412849"
server_key_version: "encryptionkey@0"
I20260928 23:54:52.830749 24387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:52.844532 24387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:52.848373 24387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:52.848604 24387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:52.857000 24387 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:43331
I20260928 23:54:52.857055 24454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:43331 every 8 connection(s)
I20260928 23:54:52.858178 24387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:52.861611 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:52.862672 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24387
I20260928 23:54:52.862882 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:52.863178 20832 external_mini_cluster.cc:1428] Setting key 994e7765cd18e07a94a8f19724ff3c29
I20260928 23:54:52.868861 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f: Bootstrap starting.
I20260928 23:54:52.871616 24455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:52.872630 24455 log.cc:824] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:54:52 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:52.875283 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f: No bootstrap required, opened a new log
I20260928 23:54:52.878948 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:52.864843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.20.88.62:43331)
0928 23:54:52.865319 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:52.865336 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:52.865373 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:54:52.866037 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:52.866057 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:52.866433 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:52.866747 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:52.867712 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:52.868986 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:52.869941 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:52.870468 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:52.870653 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:52.873800 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:52.873806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:52.873823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:52.873850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:52.875641 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:52.876197 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:52.876203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:52.876208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:52.876424 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:52.876733 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:52.876736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:52.876737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:52.877185 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:52.877357 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:52.877752 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 23:54:52.878115 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":150,"threads_started":1}
I20260928 23:54:52.880482 24455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } }
I20260928 23:54:52.880728 24455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:52.880785 24455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03076b413b164688b0d420d25be78c3f, State: Initialized, Role: FOLLOWER
I20260928 23:54:52.881233 24455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } }
I20260928 23:54:52.881412 24455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:52.881511 24455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:52.881621 24455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:52.953118 24455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } }
I20260928 23:54:52.953718 24455 leader_election.cc:302] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03076b413b164688b0d420d25be78c3f; no voters:
I20260928 23:54:52.954217 24455 leader_election.cc:288] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:52.954407 24460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:52.954689 24460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [term 1 LEADER]: Becoming Leader. State: Replica: 03076b413b164688b0d420d25be78c3f, State: Running, Role: LEADER
I20260928 23:54:52.955065 24460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } }
I20260928 23:54:52.955890 24455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:52.957020 24462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03076b413b164688b0d420d25be78c3f. Latest consensus state: current_term: 1 leader_uuid: "03076b413b164688b0d420d25be78c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } } }
I20260928 23:54:52.957067 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03076b413b164688b0d420d25be78c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03076b413b164688b0d420d25be78c3f" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43331 } } }
I20260928 23:54:52.957216 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:52.957214 24462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:52.957791 24469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:52.961361 24469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:52.967401 24469 catalog_manager.cc:1347] Generated new cluster ID: 2a6bdbae02924acf9769df7caa562156
I20260928 23:54:52.967489 24469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:52.981184 24469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:52.982275 24469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:52.988857 24469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03076b413b164688b0d420d25be78c3f: Generated new TSK 0
I20260928 23:54:52.989730 24469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
W20260928 23:54:54.376962 20832 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.88.1: real 1.381s user 0.000s sys 0.004s
I20260928 23:54:54.377599 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:54.505461 24481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:54.505779 24481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:54.505824 24481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:54.510254 24481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:54.510342 24481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:54.510406 24481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:54.516476 24481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:54.517814 24481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:54.518962 24481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:54.530684 24487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:54.530781 24486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:54.531033 24489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:54.530855 24481 server_base.cc:1034] running on GCE node
I20260928 23:54:54.531718 24481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:54.532691 24481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:54.534020 24481 hybrid_clock.cc:648] HybridClock initialized: now 1790639694533970 us; error 77 us; skew 500 ppm
Sep 28 23:54:54 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639694, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:54.538177 24481 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:54.540019 24481 webserver.cc:460] Webserver started at http://127.20.88.1:37469/ using document root <none> and password file <none>
I20260928 23:54:54.540904 24481 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:54.541002 24481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:54.541431 24481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:54.545055 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "d01daa3b614b4851a61c793cfa043606"
format_stamp: "Formatted at 2026-09-28 23:54:54 on dist-test-slave-wvpw"
server_key: "db40c55ad24319a097dfc6726daeca5a"
server_key_iv: "292d84564b4d5a61ecfdfd342a81c1af"
server_key_version: "encryptionkey@0"
I20260928 23:54:54.545861 24481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "d01daa3b614b4851a61c793cfa043606"
format_stamp: "Formatted at 2026-09-28 23:54:54 on dist-test-slave-wvpw"
server_key: "db40c55ad24319a097dfc6726daeca5a"
server_key_iv: "292d84564b4d5a61ecfdfd342a81c1af"
server_key_version: "encryptionkey@0"
I20260928 23:54:54.551532 24481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 23:54:54.555354 24496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:54.557372 24481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:54.557547 24481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "d01daa3b614b4851a61c793cfa043606"
format_stamp: "Formatted at 2026-09-28 23:54:54 on dist-test-slave-wvpw"
server_key: "db40c55ad24319a097dfc6726daeca5a"
server_key_iv: "292d84564b4d5a61ecfdfd342a81c1af"
server_key_version: "encryptionkey@0"
I20260928 23:54:54.557729 24481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:54.586345 24481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:54.590355 24481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:54.590581 24481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:54.591277 24481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:54.592276 24481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:54.592347 24481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:54.592424 24481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:54.592475 24481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:54.602746 24481 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:45421
I20260928 23:54:54.602814 24609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:45421 every 8 connection(s)
I20260928 23:54:54.604149 24481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:54.605516 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24481
I20260928 23:54:54.605665 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:54.606009 20832 external_mini_cluster.cc:1428] Setting key f16aef70f869338abdf5ec584784e070
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
Sep 28 23:54:54 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639694, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:54.720834 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:54.606502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:37651 (local address 127.20.88.62:43331)
0928 23:54:54.606799 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:54.606803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:54.607891 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 23:54:54.609594 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:54.609598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:54.609664 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:54.609790 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:54.611208 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:54.611822 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:54.613446 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:54.613723 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:54.613857 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:54.716533 (+102676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:54.716536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:54.716538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:54.716572 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:54.718297 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:54.718984 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:54.718987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:54.718988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:54.719072 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:54.719485 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:54.719488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:54.719489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:54.719702 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:54.719820 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:54.720493 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 23:54:54.720638 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
I20260928 23:54:54.721838 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:54.607141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43331 (local address 127.20.88.1:37651)
0928 23:54:54.607709 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:54.607773 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:54.609338 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:54.610004 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:54.610012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:54.610448 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:54.611026 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:54.611042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:54.611982 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:54.611987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:54.613279 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:54.613292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:54.613915 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:54.613920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:54.614136 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:54.614875 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:54.614894 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:54.716193 (+101299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:54.718481 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:54.718489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:54.718516 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:54.718848 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:54.719186 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:54.719189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:54.719191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:54.719375 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:54.719822 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:54.719841 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:54.720143 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 23:54:54.720435 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 23:54:54.720832 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":160,"threads_started":1}
I20260928 23:54:54.723359 24610 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43331
I20260928 23:54:54.723677 24610 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:54.724328 24610 heartbeater.cc:511] Master 127.20.88.62:43331 requested a full tablet report, sending...
I20260928 23:54:54.726059 24414 ts_manager.cc:194] Registered new tserver with Master: d01daa3b614b4851a61c793cfa043606 (127.20.88.1:45421)
I20260928 23:54:54.727227 24414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:37651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
W20260928 23:54:55.039131 20832 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.88.2: real 0.431s user 0.000s sys 0.004s
I20260928 23:54:55.039674 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:55.159092 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:55.159415 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:55.159462 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:55.163769 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:55.163851 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:55.163916 24616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:55.169696 24616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:55.170845 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:55.171856 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:55.182121 24622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:55.182121 24624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:55.182121 24621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:55.182497 24616 server_base.cc:1034] running on GCE node
I20260928 23:54:55.182958 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:55.183606 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:55.184762 24616 hybrid_clock.cc:648] HybridClock initialized: now 1790639695184723 us; error 51 us; skew 500 ppm
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:55.187932 24616 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:55.188952 24616 webserver.cc:460] Webserver started at http://127.20.88.2:44371/ using document root <none> and password file <none>
I20260928 23:54:55.189507 24616 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:55.189568 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:55.189752 24616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:55.191856 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "9144c5d07ec549b5986bb75ea20bfab7"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "00b1f57ce1d8d3191416b44ad750fa65"
server_key_iv: "c9c400cda6bff67ee14c4385577d9be6"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.192472 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "9144c5d07ec549b5986bb75ea20bfab7"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "00b1f57ce1d8d3191416b44ad750fa65"
server_key_iv: "c9c400cda6bff67ee14c4385577d9be6"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.196466 24616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:54:55.199070 24631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.200521 24616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:55.200664 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "9144c5d07ec549b5986bb75ea20bfab7"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "00b1f57ce1d8d3191416b44ad750fa65"
server_key_iv: "c9c400cda6bff67ee14c4385577d9be6"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.200807 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:55.220583 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:55.224023 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:55.224290 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:55.224915 24616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:55.225761 24616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:55.225833 24616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.225906 24616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:55.225957 24616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.234781 24616 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:45291
I20260928 23:54:55.234800 24744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:45291 every 8 connection(s)
I20260928 23:54:55.235782 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:55.236945 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24616
I20260928 23:54:55.237074 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:55.237427 20832 external_mini_cluster.cc:1428] Setting key 2a9bdf56cbf2f9333e3c9e60fd7ad04f
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:54:55.251396 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.237855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38655 (local address 127.20.88.62:43331)
0928 23:54:55.238253 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:55.238259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:55.239038 (+ 779us) server_negotiation.cc:407] Connection header received
0928 23:54:55.240570 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:55.240573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:55.240633 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:55.240753 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:55.241987 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.242535 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.243728 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.243972 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.244091 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.247023 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:55.247027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:55.247029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:55.247057 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:55.248669 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.249345 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.249348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.249349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.249435 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.249967 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.249971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.249972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.250216 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:55.250465 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:55.251046 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 23:54:55.251195 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260928 23:54:55.252409 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.238412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43331 (local address 127.20.88.2:38655)
0928 23:54:55.238897 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:55.238953 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:55.240345 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:55.240946 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:55.240954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:55.241332 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:55.241827 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:55.241842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.242670 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.242673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:55.243586 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:55.243595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.244138 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.244142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:55.244438 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.245113 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:55.245130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:55.246832 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:55.248892 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:55.248900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:55.248925 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:55.249235 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:55.249584 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:55.249588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:55.249592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:55.249823 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:55.250465 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:55.250474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:55.250786 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 23:54:55.251004 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 23:54:55.251418 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":146,"threads_started":1}
I20260928 23:54:55.253830 24745 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43331
I20260928 23:54:55.254140 24745 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:55.254874 24745 heartbeater.cc:511] Master 127.20.88.62:43331 requested a full tablet report, sending...
I20260928 23:54:55.256313 24414 ts_manager.cc:194] Registered new tserver with Master: 9144c5d07ec549b5986bb75ea20bfab7 (127.20.88.2:45291)
I20260928 23:54:55.257038 24414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:38655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:55.268467 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:55.394696 24751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:55.395058 24751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:55.395139 24751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:55.399446 24751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:55.399588 24751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:55.399693 24751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:55.405658 24751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:55.406919 24751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:55.408069 24751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:55.419363 24756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:55.419364 24759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:55.419400 24757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:55.419916 24751 server_base.cc:1034] running on GCE node
I20260928 23:54:55.420475 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:55.421226 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:55.422408 24751 hybrid_clock.cc:648] HybridClock initialized: now 1790639695422386 us; error 33 us; skew 500 ppm
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:55.425590 24751 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:55.426519 24751 webserver.cc:460] Webserver started at http://127.20.88.3:46277/ using document root <none> and password file <none>
I20260928 23:54:55.427094 24751 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:55.427168 24751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:55.427347 24751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:55.429663 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "a877db853b434427b9daa17366c0ff1f"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "f56183345cff8fd07c74f61eb00617c8"
server_key_iv: "10031d8179be5f7777105327234faa63"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.430238 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "a877db853b434427b9daa17366c0ff1f"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "f56183345cff8fd07c74f61eb00617c8"
server_key_iv: "10031d8179be5f7777105327234faa63"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.434464 24751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:54:55.437347 24766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.438833 24751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:55.438979 24751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "a877db853b434427b9daa17366c0ff1f"
format_stamp: "Formatted at 2026-09-28 23:54:55 on dist-test-slave-wvpw"
server_key: "f56183345cff8fd07c74f61eb00617c8"
server_key_iv: "10031d8179be5f7777105327234faa63"
server_key_version: "encryptionkey@0"
I20260928 23:54:55.439110 24751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:55.460139 24751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:55.463627 24751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:55.463801 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:55.464536 24751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:55.465531 24751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:55.465606 24751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.465673 24751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:55.465698 24751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:55.475359 24751 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:45665
I20260928 23:54:55.475379 24879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:45665 every 8 connection(s)
I20260928 23:54:55.476504 24751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:55.486196 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24751
I20260928 23:54:55.486363 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:55.486717 20832 external_mini_cluster.cc:1428] Setting key df4ba91e76d5a5fa565edc349a2c3de2
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:55.491385 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.478644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44641 (local address 127.20.88.62:43331)
0928 23:54:55.478833 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:55.478837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:55.479703 (+ 866us) server_negotiation.cc:407] Connection header received
0928 23:54:55.481212 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:55.481215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:55.481272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:55.481353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:55.482584 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.483131 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.484380 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.484655 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.484754 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.487641 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:55.487646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:55.487649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:55.487685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:55.489401 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.489907 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.489909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.489910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.489987 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.490324 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.490327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.490328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.490557 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:55.490648 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:55.491141 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 23:54:55.491259 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 23:54:55.492292 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.479024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43331 (local address 127.20.88.3:44641)
0928 23:54:55.479516 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:55.479563 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:55.480973 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:55.481538 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:55.481545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:55.481921 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:55.482428 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:55.482442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.483282 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.483285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:55.484185 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:55.484195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.484809 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.484812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:55.484959 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.485579 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:55.485596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:55.487450 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:55.489510 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:55.489515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:55.489530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:55.489814 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:55.490064 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:55.490066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:55.490067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:55.490213 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:55.490660 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:55.490665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:55.490918 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 23:54:55.491082 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 23:54:55.491366 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260928 23:54:55.493556 24880 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43331
I20260928 23:54:55.493821 24880 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:55.494339 24880 heartbeater.cc:511] Master 127.20.88.62:43331 requested a full tablet report, sending...
I20260928 23:54:55.495391 24414 ts_manager.cc:194] Registered new tserver with Master: a877db853b434427b9daa17366c0ff1f (127.20.88.3:45665)
I20260928 23:54:55.495998 24414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:44641
I20260928 23:54:55.500975 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:54:55.528213 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.518137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.20.88.1:45421)
0928 23:54:55.518456 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:55.518462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:55.518507 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:54:55.518558 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:55.518564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:55.518761 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:55.518888 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:55.519626 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.520496 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.521347 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.521782 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.521873 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.524509 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:55.524519 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:55.524535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:55.524585 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:55.526193 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.526654 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.526678 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.526679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.526761 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.527072 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.527074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.527075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.527330 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:55.527499 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:55.527832 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 23:54:55.528049 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:55.540019 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:55.531432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.20.88.62:43331)
0928 23:54:55.531603 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:55.531607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:55.531621 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:54:55.531736 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:55.531738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:55.531780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:55.531849 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:55.532535 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.533062 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.533850 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:55.534064 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:55.534159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:55.536778 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:55.536781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:55.536783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:55.536813 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:55.538160 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.538637 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.538640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.538641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.538725 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:55.539145 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:55.539147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:55.539149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:55.539391 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:55.539503 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:55.539714 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 23:54:55.539819 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 23:54:55.540486 24414 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34344
I20260928 23:54:55.541186 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24481
I20260928 23:54:55.547016 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24616
I20260928 23:54:55.553153 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24751
I20260928 23:54:55.559195 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24387
2026-09-28T23:54:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (6008 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24892](info): set up 2 sockets
Sep 28 23:54:55 dist-test-slave-wvpw krb5kdc[24892](info): commencing operation
krb5kdc: starting...
W20260928 23:54:56.604941 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:54:56 dist-test-slave-wvpw krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:54:56.736712 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:33389
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:33389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:56.852867 24905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:56.853223 24905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:56.853308 24905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:56.857704 24905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:56.857811 24905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:56.857848 24905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:56.857887 24905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:56.857924 24905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:56.863610 24905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:33389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:56.864923 24905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:56.866044 24905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:56.875679 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:56.875717 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:56.875717 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:56.875967 24905 server_base.cc:1034] running on GCE node
I20260928 23:54:56.876778 24905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:56.878196 24905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:56.879421 24905 hybrid_clock.cc:648] HybridClock initialized: now 1790639696879402 us; error 35 us; skew 500 ppm
Sep 28 23:54:56 dist-test-slave-wvpw krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:56.882699 24905 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:54:56.883586 24905 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:56.883658 24905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:56.883894 24905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:56.886219 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "41794f40009e4139ba95350fe644491d"
format_stamp: "Formatted at 2026-09-28 23:54:56 on dist-test-slave-wvpw"
server_key: "0172a6eabde74e9b82680857e9740293"
server_key_iv: "02fb8d86fe49840e464d67586f7db385"
server_key_version: "encryptionkey@0"
I20260928 23:54:56.886847 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "41794f40009e4139ba95350fe644491d"
format_stamp: "Formatted at 2026-09-28 23:54:56 on dist-test-slave-wvpw"
server_key: "0172a6eabde74e9b82680857e9740293"
server_key_iv: "02fb8d86fe49840e464d67586f7db385"
server_key_version: "encryptionkey@0"
I20260928 23:54:56.891237 24905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 23:54:56.894166 24919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:56.895694 24905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:54:56.895910 24905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "41794f40009e4139ba95350fe644491d"
format_stamp: "Formatted at 2026-09-28 23:54:56 on dist-test-slave-wvpw"
server_key: "0172a6eabde74e9b82680857e9740293"
server_key_iv: "02fb8d86fe49840e464d67586f7db385"
server_key_version: "encryptionkey@0"
I20260928 23:54:56.896059 24905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:56.912173 24905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:56.916055 24905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:56.916400 24905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:56.924360 24905 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:33389
I20260928 23:54:56.924666 24971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:33389 every 8 connection(s)
I20260928 23:54:56.925165 24905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:56.928501 24972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:56.934196 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24905
I20260928 23:54:56.934363 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:56.934644 20832 external_mini_cluster.cc:1428] Setting key 2b588cc097cd64b1a842227dc35e28b9
I20260928 23:54:56.936864 24972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: Bootstrap starting.
I20260928 23:54:56.939404 24972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:56.940374 24972 log.cc:824] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:56.942924 24972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: No bootstrap required, opened a new log
Sep 28 23:54:56 dist-test-slave-wvpw krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:56.947190 24972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } }
I20260928 23:54:56.947516 24972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:56.947608 24972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41794f40009e4139ba95350fe644491d, State: Initialized, Role: FOLLOWER
I20260928 23:54:56.948207 24972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } }
I20260928 23:54:56.948447 24972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:56.948556 24972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:56.948673 24972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:56.949824 24972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } }
I20260928 23:54:56.950253 24972 leader_election.cc:302] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41794f40009e4139ba95350fe644491d; no voters:
I20260928 23:54:56.950728 24972 leader_election.cc:288] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:56.950824 24977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:56.950970 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:56.936188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.20.88.62:33389)
0928 23:54:56.936733 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:56.936750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:56.936790 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:54:56.937548 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:56.937568 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:56.937950 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:56.938345 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:56.939199 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:56.940409 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:56.941585 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:56.942143 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:56.942272 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:56.945392 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:56.945398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:56.945415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:56.945448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:56.947359 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:56.947925 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:56.947932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:56.947937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:56.948060 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:56.948444 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:56.948451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:56.948453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:56.949025 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:56.949252 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:56.949563 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 23:54:56.949891 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":161,"threads_started":1}
I20260928 23:54:56.951056 24977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [term 1 LEADER]: Becoming Leader. State: Replica: 41794f40009e4139ba95350fe644491d, State: Running, Role: LEADER
I20260928 23:54:56.951411 24977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } }
I20260928 23:54:56.952307 24972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:56.953365 24978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41794f40009e4139ba95350fe644491d. Latest consensus state: current_term: 1 leader_uuid: "41794f40009e4139ba95350fe644491d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } } }
I20260928 23:54:56.953658 24978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:56.954042 24979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41794f40009e4139ba95350fe644491d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41794f40009e4139ba95350fe644491d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33389 } } }
I20260928 23:54:56.954144 24979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d [sys.catalog]: This master's current role is: LEADER
W20260928 23:54:56.957615 24993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:56.957698 24993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:56.957787 24986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:56.958563 24986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:56.965334 24986 catalog_manager.cc:1347] Generated new cluster ID: 9dfcd53b2e1c4cc6aafe8a8f97026eee
I20260928 23:54:56.965438 24986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:56.988278 24986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:56.989768 24986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:56.997511 24986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: Generated new TSK 0
I20260928 23:54:56.998358 24986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:54:57.032625 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:57.153661 24998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:57.154001 24998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:57.154080 24998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:57.158663 24998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:57.158794 24998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:57.158931 24998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:57.165319 24998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:57.166445 24998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:57.167487 24998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:57.178165 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:57.178165 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:57.178246 25006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:57.178979 24998 server_base.cc:1034] running on GCE node
I20260928 23:54:57.179526 24998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:57.180316 24998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:57.181494 24998 hybrid_clock.cc:648] HybridClock initialized: now 1790639697181482 us; error 31 us; skew 500 ppm
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:57.184645 24998 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:54:57.185503 24998 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:57.185567 24998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:57.185791 24998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:57.187983 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "00d6a1a6ffc847a0be2a47c2327ce478"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "e969501c632c3e74adc9b631bc1e677d"
server_key_iv: "3c12c6b07affbb365bc226824d34989c"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.188627 24998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "00d6a1a6ffc847a0be2a47c2327ce478"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "e969501c632c3e74adc9b631bc1e677d"
server_key_iv: "3c12c6b07affbb365bc226824d34989c"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.192643 24998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:54:57.195122 25012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.196574 24998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:57.196736 24998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "00d6a1a6ffc847a0be2a47c2327ce478"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "e969501c632c3e74adc9b631bc1e677d"
server_key_iv: "3c12c6b07affbb365bc226824d34989c"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.196871 24998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:57.212016 24998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:57.215628 24998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:57.215848 24998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:57.216552 24998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:57.217437 24998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:57.217509 24998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.217582 24998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:57.217635 24998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.227415 24998 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39325
I20260928 23:54:57.227432 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39325 every 8 connection(s)
I20260928 23:54:57.228060 24998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:57.230289 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 24998
I20260928 23:54:57.230434 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:57.230748 20832 external_mini_cluster.cc:1428] Setting key c3437a364906145e87e39c1b96344d57
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:54:57.244366 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.230005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:39655 (local address 127.20.88.62:33389)
0928 23:54:57.230181 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.230187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.231213 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 23:54:57.232810 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.232816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.232875 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.232973 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:57.234450 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.234988 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.236268 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.236711 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.236824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.239527 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:57.239535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:57.239540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:57.239580 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:57.241380 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.241996 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.241999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.242000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.242105 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.242557 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.242566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.242567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.242783 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:57.242886 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.243924 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.244124 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:54:57.245347 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.230444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33389 (local address 127.20.88.1:39655)
0928 23:54:57.231006 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.231067 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.232577 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.233182 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.233190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.233582 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:57.234279 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.234303 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.235134 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.235137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.236095 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.236113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.236885 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.236888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.237049 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.237749 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:57.237766 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:57.239314 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:57.241558 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.241564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.241582 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.241889 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.242224 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.242226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.242228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.242386 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.242947 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:57.242957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:57.243613 (+ 656us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.243869 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.244369 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":150,"threads_started":1}
I20260928 23:54:57.246917 25126 heartbeater.cc:347] Connected to a master server at 127.20.88.62:33389
I20260928 23:54:57.247222 25126 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:57.247818 25126 heartbeater.cc:511] Master 127.20.88.62:33389 requested a full tablet report, sending...
I20260928 23:54:57.249909 24936 ts_manager.cc:194] Registered new tserver with Master: 00d6a1a6ffc847a0be2a47c2327ce478 (127.20.88.1:39325)
I20260928 23:54:57.251670 24936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:39655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:54:57.264143 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:57.383417 25131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:57.383709 25131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:57.383756 25131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:57.388764 25131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:57.388839 25131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:57.388914 25131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:57.394802 25131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:57.395948 25131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:57.397110 25131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:57.408528 25139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:57.408563 25136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:57.408628 25131 server_base.cc:1034] running on GCE node
W20260928 23:54:57.408529 25137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:57.409216 25131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:57.410018 25131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:57.411209 25131 hybrid_clock.cc:648] HybridClock initialized: now 1790639697411160 us; error 67 us; skew 500 ppm
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:57.414381 25131 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:54:57.415246 25131 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:57.415318 25131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:57.415555 25131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:57.417876 25131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "052faf470ab94af39100223254fbbbaf"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "73d3554186da86f19c30d74555d4ee49"
server_key_iv: "e27c28f82564b33e367006324c393145"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.418526 25131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "052faf470ab94af39100223254fbbbaf"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "73d3554186da86f19c30d74555d4ee49"
server_key_iv: "e27c28f82564b33e367006324c393145"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.422770 25131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:57.425629 25145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.427074 25131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:57.427217 25131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "052faf470ab94af39100223254fbbbaf"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "73d3554186da86f19c30d74555d4ee49"
server_key_iv: "e27c28f82564b33e367006324c393145"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.427525 25131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:57.458225 25131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:57.462056 25131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:57.462309 25131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:57.462975 25131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:57.463933 25131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:57.463982 25131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.464026 25131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:57.464066 25131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.474133 25131 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:36863
I20260928 23:54:57.474149 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:36863 every 8 connection(s)
I20260928 23:54:57.474826 25131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:57.481781 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25131
I20260928 23:54:57.481926 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:57.482278 20832 external_mini_cluster.cc:1428] Setting key 59f97f6bacf0acdbb61afd6f7ffec463
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:57.491668 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.476824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50345 (local address 127.20.88.62:33389)
0928 23:54:57.477007 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.477011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.478026 (+ 1015us) server_negotiation.cc:407] Connection header received
0928 23:54:57.479436 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.479439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.479505 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.479577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:57.481073 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.482024 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.483658 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.484086 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.484201 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.487031 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:57.487035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:57.487037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:57.487066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:57.488794 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.489591 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.489595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.489596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.489686 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.490199 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.490203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.490204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.490426 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:57.490522 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.491349 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.491489 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:54:57.493008 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.477261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33389 (local address 127.20.88.2:50345)
0928 23:54:57.477801 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.477851 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.479176 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.479773 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.479780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.480198 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:57.480905 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.480924 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.482162 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.482171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.483511 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.483524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.484305 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.484316 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.484494 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.485194 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:57.485212 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:57.486818 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:57.488950 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.488960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.488984 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.489445 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.489797 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.489803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.489808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.490058 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.490554 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:57.490563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:57.491014 (+ 451us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.491279 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.491720 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":128,"threads_started":1}
I20260928 23:54:57.494616 25259 heartbeater.cc:347] Connected to a master server at 127.20.88.62:33389
I20260928 23:54:57.494866 25259 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:57.495332 25259 heartbeater.cc:511] Master 127.20.88.62:33389 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:54:57.496510 24936 ts_manager.cc:194] Registered new tserver with Master: 052faf470ab94af39100223254fbbbaf (127.20.88.2:36863)
I20260928 23:54:57.497239 24936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:50345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:54:57.516198 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:57.637183 25264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:57.637495 25264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:57.637552 25264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:57.641819 25264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:57.641896 25264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:57.641994 25264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:57.647861 25264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:57.649058 25264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:57.650175 25264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:57.660585 25272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:57.660542 25270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:57.660542 25269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:57.661065 25264 server_base.cc:1034] running on GCE node
I20260928 23:54:57.661476 25264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:57.662132 25264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:57.663316 25264 hybrid_clock.cc:648] HybridClock initialized: now 1790639697663287 us; error 49 us; skew 500 ppm
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:57.666361 25264 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:54:57.667204 25264 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:57.667311 25264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:57.667563 25264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:57.669925 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "0d3312ab607d45e1869b89bc2a3c8de6"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "6fb963fda716f66bea1c480f691c4ab7"
server_key_iv: "66bb7f2fa2cd07e3efb8fdd96b715eea"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.670567 25264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "0d3312ab607d45e1869b89bc2a3c8de6"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "6fb963fda716f66bea1c480f691c4ab7"
server_key_iv: "66bb7f2fa2cd07e3efb8fdd96b715eea"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.674746 25264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:57.677767 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.679265 25264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:54:57.679544 25264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "0d3312ab607d45e1869b89bc2a3c8de6"
format_stamp: "Formatted at 2026-09-28 23:54:57 on dist-test-slave-wvpw"
server_key: "6fb963fda716f66bea1c480f691c4ab7"
server_key_iv: "66bb7f2fa2cd07e3efb8fdd96b715eea"
server_key_version: "encryptionkey@0"
I20260928 23:54:57.679693 25264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:57.696635 25264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:57.699901 25264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:57.700134 25264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:57.700855 25264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:57.701828 25264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:57.701900 25264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.701978 25264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:57.702032 25264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:57.711231 25264 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:45903
I20260928 23:54:57.711267 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:45903 every 8 connection(s)
I20260928 23:54:57.711885 25264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:57.713649 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25264
I20260928 23:54:57.713824 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:57.714176 20832 external_mini_cluster.cc:1428] Setting key 459349d78d3cdc41c03662254336609d
Sep 28 23:54:57 dist-test-slave-wvpw krb5kdc[24892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639697, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:54:57.727325 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.713931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:59941 (local address 127.20.88.62:33389)
0928 23:54:57.714124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.714129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.715264 (+ 1135us) server_negotiation.cc:407] Connection header received
0928 23:54:57.716965 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.716969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.717025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.717108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:57.718363 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.718904 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.720119 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.720414 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.720529 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.723399 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:57.723403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:57.723405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:57.723432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:57.725008 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.725670 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.725674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.725676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.725783 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.726162 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.726164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.726165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.726366 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:57.726447 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.727013 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.727157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:54:57.728407 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.714470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33389 (local address 127.20.88.3:59941)
0928 23:54:57.715103 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.715167 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.716748 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.717332 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.717340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.717708 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:57.718215 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.718229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.719041 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.719044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.719976 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.719987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.720576 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.720580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.720770 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.721431 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:57.721448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:57.723157 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:57.725140 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.725145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.725160 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.725437 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.725897 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:57.725899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:57.725901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:57.726056 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:57.726461 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:57.726467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:57.726743 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.726968 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.727339 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":177,"threads_started":1}
I20260928 23:54:57.729641 25392 heartbeater.cc:347] Connected to a master server at 127.20.88.62:33389
I20260928 23:54:57.729861 25392 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:57.730293 25392 heartbeater.cc:511] Master 127.20.88.62:33389 requested a full tablet report, sending...
I20260928 23:54:57.731324 24936 ts_manager.cc:194] Registered new tserver with Master: 0d3312ab607d45e1869b89bc2a3c8de6 (127.20.88.3:45903)
I20260928 23:54:57.732048 24936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:59941
I20260928 23:54:57.740931 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:54:57.751477 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.743020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.20.88.62:33389)
0928 23:54:57.743201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.743205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.743342 (+ 137us) server_negotiation.cc:407] Connection header received
0928 23:54:57.743522 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.743525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.743576 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.743664 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:57.744445 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.744979 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.745947 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.746184 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.746299 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.748037 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:57.748040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:57.748042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:57.748071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:57.749585 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.750079 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.750083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.750085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.750192 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:57.750486 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:57.750490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:57.750492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:57.750762 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:57.750911 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.751164 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.751306 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 23:54:57.755049 24936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:57.758301 24936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:57.773765 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.768530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:45903 (local address 127.0.0.1:45948)
0928 23:54:57.769056 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.769113 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.769255 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.769831 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.769833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.769858 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:57.770026 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.770035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.771046 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.771053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.772413 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.772423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.773310 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.773313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.773392 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.773424 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.773524 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.773632 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":81,"spinlock_wait_cycles":8064,"thread_start_us":119,"threads_started":1}
I20260928 23:54:57.773883 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.768556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.20.88.3:45903)
0928 23:54:57.769430 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.769442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.769464 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:54:57.769527 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.769531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.769687 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.769971 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:57.770118 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.770912 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.772529 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.773188 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.773299 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.773450 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.773569 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.773721 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260928 23:54:57.775885 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.767701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39325 (local address 127.0.0.1:42958)
0928 23:54:57.769056 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.769113 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.769255 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.769969 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.769971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.769984 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:57.770110 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.770116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.771502 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.771505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.772903 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.772914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.775438 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.775441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.775528 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.775552 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.775644 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.775729 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":220,"spinlock_wait_cycles":2176,"thread_start_us":78,"threads_started":1}
I20260928 23:54:57.775959 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.767824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.20.88.1:39325)
0928 23:54:57.768437 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.768445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.769233 (+ 788us) server_negotiation.cc:407] Connection header received
0928 23:54:57.769453 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.769459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.769656 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.769786 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:57.770239 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.771365 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.773045 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.773905 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.774019 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.774184 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.775700 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.775809 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":135,"threads_started":1}
I20260928 23:54:57.777221 25326 tablet_service.cc:1467] Processing CreateTablet for tablet bd99b13a460e416b84931323a6dd42eb (DEFAULT_TABLE table=test-table [id=5d74c02fdad0456999397e620f3e07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:57.778275 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99b13a460e416b84931323a6dd42eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:57.778514 25060 tablet_service.cc:1467] Processing CreateTablet for tablet bd99b13a460e416b84931323a6dd42eb (DEFAULT_TABLE table=test-table [id=5d74c02fdad0456999397e620f3e07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:57.779006 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.767701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36863 (local address 127.0.0.1:51872)
0928 23:54:57.769057 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:57.769112 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:57.769260 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:57.769776 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:57.769782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:57.769803 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:57.769992 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.770003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.771505 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.771507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.772912 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:57.772918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.778545 (+ 5627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.778550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:57.778656 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.778683 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:54:57.778770 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:54:57.778853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":186,"spinlock_wait_cycles":2432,"thread_start_us":85,"threads_started":1}
I20260928 23:54:57.779546 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:57.767957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.20.88.2:36863)
0928 23:54:57.768671 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:57.768683 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:57.769291 (+ 608us) server_negotiation.cc:407] Connection header received
0928 23:54:57.769424 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:57.769434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:57.769610 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:57.769783 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:57.770136 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.771343 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.774254 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:57.775088 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:57.775216 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:57.775365 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:57.779247 (+ 3882us) server_negotiation.cc:299] Negotiation successful
0928 23:54:57.779379 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":125,"threads_started":1}
I20260928 23:54:57.779645 25060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99b13a460e416b84931323a6dd42eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:57.782191 25193 tablet_service.cc:1467] Processing CreateTablet for tablet bd99b13a460e416b84931323a6dd42eb (DEFAULT_TABLE table=test-table [id=5d74c02fdad0456999397e620f3e07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:57.783246 25193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd99b13a460e416b84931323a6dd42eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:57.786768 25409 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Bootstrap starting.
I20260928 23:54:57.789018 25410 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Bootstrap starting.
I20260928 23:54:57.789700 25409 tablet_bootstrap.cc:654] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:57.790652 25411 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Bootstrap starting.
I20260928 23:54:57.790723 25409 log.cc:824] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:57.791393 25410 tablet_bootstrap.cc:654] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:57.792835 25410 log.cc:824] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:57.794176 25409 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: No bootstrap required, opened a new log
I20260928 23:54:57.794194 25411 tablet_bootstrap.cc:654] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:57.794421 25409 ts_tablet_manager.cc:1397] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 23:54:57.795032 25410 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: No bootstrap required, opened a new log
I20260928 23:54:57.795051 25411 log.cc:824] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:57.795231 25410 ts_tablet_manager.cc:1397] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:57.797281 25411 tablet_bootstrap.cc:492] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: No bootstrap required, opened a new log
I20260928 23:54:57.797485 25411 ts_tablet_manager.cc:1397] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 23:54:57.799144 25410 raft_consensus.cc:348] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.799155 25409 raft_consensus.cc:348] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.799468 25410 raft_consensus.cc:374] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:57.799474 25409 raft_consensus.cc:374] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:57.799528 25410 raft_consensus.cc:729] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3312ab607d45e1869b89bc2a3c8de6, State: Initialized, Role: FOLLOWER
I20260928 23:54:57.799547 25409 raft_consensus.cc:729] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d6a1a6ffc847a0be2a47c2327ce478, State: Initialized, Role: FOLLOWER
I20260928 23:54:57.800032 25410 consensus_queue.cc:260] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.800024 25409 consensus_queue.cc:260] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.800954 25392 heartbeater.cc:503] Master 127.20.88.62:33389 was elected leader, sending a full tablet report...
I20260928 23:54:57.801258 25126 heartbeater.cc:503] Master 127.20.88.62:33389 was elected leader, sending a full tablet report...
I20260928 23:54:57.801908 25411 raft_consensus.cc:348] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.802239 25411 raft_consensus.cc:374] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:57.802301 25411 raft_consensus.cc:729] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052faf470ab94af39100223254fbbbaf, State: Initialized, Role: FOLLOWER
I20260928 23:54:57.802527 25410 ts_tablet_manager.cc:1428] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:54:57.803625 25411 consensus_queue.cc:260] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:57.804805 25411 ts_tablet_manager.cc:1428] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:54:57.804816 25259 heartbeater.cc:503] Master 127.20.88.62:33389 was elected leader, sending a full tablet report...
I20260928 23:54:57.805176 25409 ts_tablet_manager.cc:1428] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
W20260928 23:54:57.962874 25393 tablet.cc:2369] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:54:57.976040 25260 tablet.cc:2369] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:54:57.979450 25127 tablet.cc:2369] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:58.039119 25415 raft_consensus.cc:482] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:58.039479 25415 raft_consensus.cc:504] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:58.040968 25415 leader_election.cc:288] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 052faf470ab94af39100223254fbbbaf (127.20.88.2:36863), 0d3312ab607d45e1869b89bc2a3c8de6 (127.20.88.3:45903)
I20260928 23:54:58.045339 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.041343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:36527 (local address 127.20.88.3:45903)
0928 23:54:58.041622 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.041627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.041761 (+ 134us) server_negotiation.cc:407] Connection header received
0928 23:54:58.041916 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.041919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.042019 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.042121 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:58.042425 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.042932 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044199 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.044677 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044802 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:58.044875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.045125 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.045200 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260928 23:54:58.045341 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.041256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:43971 (local address 127.20.88.2:36863)
0928 23:54:58.041409 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.041416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.041699 (+ 283us) server_negotiation.cc:407] Connection header received
0928 23:54:58.041867 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.041870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.041923 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.042020 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:54:58.042427 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.042933 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044194 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.044679 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044804 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:58.044873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.045149 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.045234 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:54:58.045368 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.041238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:45903 (local address 127.20.88.1:36527)
0928 23:54:58.041648 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.041663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.041800 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.042124 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.042127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.042165 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:58.042303 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:58.042308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.043067 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.043071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:58.044056 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:58.044065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.044812 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:58.044964 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:58.044997 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.045068 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.045176 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":109,"spinlock_wait_cycles":768,"thread_start_us":116,"threads_started":1}
I20260928 23:54:58.045367 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.041236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36863 (local address 127.20.88.1:43971)
0928 23:54:58.041570 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.041587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.041750 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.042057 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.042071 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.042093 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:54:58.042306 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:58.042312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.043057 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.043060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:58.044054 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:58.044065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.044812 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.044816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:58.044998 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:58.045017 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.045114 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.045195 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":101,"spinlock_wait_cycles":9472,"thread_start_us":148,"threads_started":1}
I20260928 23:54:58.046283 25346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99b13a460e416b84931323a6dd42eb" candidate_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" is_pre_election: true
I20260928 23:54:58.046484 25213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99b13a460e416b84931323a6dd42eb" candidate_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052faf470ab94af39100223254fbbbaf" is_pre_election: true
I20260928 23:54:58.046700 25346 raft_consensus.cc:2454] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d6a1a6ffc847a0be2a47c2327ce478 in term 0.
I20260928 23:54:58.046810 25213 raft_consensus.cc:2454] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d6a1a6ffc847a0be2a47c2327ce478 in term 0.
I20260928 23:54:58.047299 25013 leader_election.cc:302] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00d6a1a6ffc847a0be2a47c2327ce478, 052faf470ab94af39100223254fbbbaf, 0d3312ab607d45e1869b89bc2a3c8de6; no voters:
I20260928 23:54:58.047608 25415 raft_consensus.cc:2781] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:58.047689 25415 raft_consensus.cc:482] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:58.047720 25415 raft_consensus.cc:3037] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:58.048895 25415 raft_consensus.cc:504] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:58.049332 25415 leader_election.cc:288] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [CANDIDATE]: Term 1 election: Requested vote from peers 052faf470ab94af39100223254fbbbaf (127.20.88.2:36863), 0d3312ab607d45e1869b89bc2a3c8de6 (127.20.88.3:45903)
I20260928 23:54:58.049790 25346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99b13a460e416b84931323a6dd42eb" candidate_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3312ab607d45e1869b89bc2a3c8de6"
I20260928 23:54:58.049800 25213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd99b13a460e416b84931323a6dd42eb" candidate_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052faf470ab94af39100223254fbbbaf"
I20260928 23:54:58.049939 25346 raft_consensus.cc:3037] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:58.049953 25213 raft_consensus.cc:3037] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:58.051062 25213 raft_consensus.cc:2454] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d6a1a6ffc847a0be2a47c2327ce478 in term 1.
I20260928 23:54:58.051077 25346 raft_consensus.cc:2454] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d6a1a6ffc847a0be2a47c2327ce478 in term 1.
I20260928 23:54:58.051434 25013 leader_election.cc:302] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d6a1a6ffc847a0be2a47c2327ce478, 0d3312ab607d45e1869b89bc2a3c8de6; no voters:
I20260928 23:54:58.051648 25415 raft_consensus.cc:2781] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:58.051908 25415 raft_consensus.cc:686] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 1 LEADER]: Becoming Leader. State: Replica: 00d6a1a6ffc847a0be2a47c2327ce478, State: Running, Role: LEADER
I20260928 23:54:58.052321 25415 consensus_queue.cc:237] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } }
I20260928 23:54:58.056180 24934 catalog_manager.cc:5495] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00d6a1a6ffc847a0be2a47c2327ce478 (127.20.88.1). New cstate: current_term: 1 leader_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00d6a1a6ffc847a0be2a47c2327ce478" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 } health_report { overall_health: UNKNOWN } } }
I20260928 23:54:58.070015 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.066411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.20.88.1:39325)
0928 23:54:58.066584 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.066588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.066616 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:54:58.066693 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.066695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.066749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.066834 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:58.067089 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.067631 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.068559 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:58.068815 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:58.068908 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:58.069177 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:58.069641 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:58.069783 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.069832 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.069893 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:54:58.076642 25346 raft_consensus.cc:1261] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 1 FOLLOWER]: Refusing update from remote peer 00d6a1a6ffc847a0be2a47c2327ce478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:58.076808 25213 raft_consensus.cc:1261] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 1 FOLLOWER]: Refusing update from remote peer 00d6a1a6ffc847a0be2a47c2327ce478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:58.077550 25415 consensus_queue.cc:1035] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3312ab607d45e1869b89bc2a3c8de6" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:58.077848 25420 consensus_queue.cc:1035] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "052faf470ab94af39100223254fbbbaf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:58.087486 25425 mvcc.cc:205] Tried to move back new op lower bound from 7334460203312975872 to 7334460203225632768. Current Snapshot: MvccSnapshot[applied={T|T < 7334460203312975872}]
I20260928 23:54:58.087153 25424 mvcc.cc:205] Tried to move back new op lower bound from 7334460203312975872 to 7334460203225632768. Current Snapshot: MvccSnapshot[applied={T|T < 7334460203312975872}]
I20260928 23:54:58.097072 24936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:58.097273 24936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:58.100005 24936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41794f40009e4139ba95350fe644491d: Sending DeleteTablet for 3 replicas of tablet bd99b13a460e416b84931323a6dd42eb
I20260928 23:54:58.101022 25326 tablet_service.cc:1514] Processing DeleteTablet for tablet bd99b13a460e416b84931323a6dd42eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:58 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:45948
I20260928 23:54:58.101018 25060 tablet_service.cc:1514] Processing DeleteTablet for tablet bd99b13a460e416b84931323a6dd42eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:58 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:42958
I20260928 23:54:58.101089 25193 tablet_service.cc:1514] Processing DeleteTablet for tablet bd99b13a460e416b84931323a6dd42eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:58 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:51872
I20260928 23:54:58.101646 25433 tablet_replica.cc:330] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: stopping tablet replica
I20260928 23:54:58.101646 25434 tablet_replica.cc:330] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478: stopping tablet replica
I20260928 23:54:58.102018 25434 raft_consensus.cc:2229] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:54:58.102023 25433 raft_consensus.cc:2229] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:58.102371 25433 raft_consensus.cc:2258] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:58.102419 25434 raft_consensus.cc:2258] T bd99b13a460e416b84931323a6dd42eb P 00d6a1a6ffc847a0be2a47c2327ce478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:58.102694 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24998
I20260928 23:54:58.103958 25433 ts_tablet_manager.cc:1905] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:58.104825 25435 tablet_replica.cc:330] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: stopping tablet replica
I20260928 23:54:58.105216 25435 raft_consensus.cc:2229] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:58.105583 25435 raft_consensus.cc:2258] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:58.106758 25433 ts_tablet_manager.cc:1918] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:58.106878 25433 log.cc:1196] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/bd99b13a460e416b84931323a6dd42eb
I20260928 23:54:58.107201 25433 ts_tablet_manager.cc:1939] T bd99b13a460e416b84931323a6dd42eb P 0d3312ab607d45e1869b89bc2a3c8de6: Deleting consensus metadata
I20260928 23:54:58.107218 25435 ts_tablet_manager.cc:1905] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:58.108479 24920 catalog_manager.cc:4841] TS 0d3312ab607d45e1869b89bc2a3c8de6 (127.20.88.3:45903): tablet bd99b13a460e416b84931323a6dd42eb (table test-table [id=5d74c02fdad0456999397e620f3e07cd]) successfully deleted
I20260928 23:54:58.110870 25435 ts_tablet_manager.cc:1918] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:58.111001 25435 log.cc:1196] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/bd99b13a460e416b84931323a6dd42eb
I20260928 23:54:58.111343 25435 ts_tablet_manager.cc:1939] T bd99b13a460e416b84931323a6dd42eb P 052faf470ab94af39100223254fbbbaf: Deleting consensus metadata
W20260928 23:54:58.111620 24922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:39325 (error 108)
I20260928 23:54:58.111667 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25131
I20260928 23:54:58.113539 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.113243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39325 (local address 127.0.0.1:42980)
0928 23:54:58.113380 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.113460 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:54:58.113819 24922 catalog_manager.cc:4584] TS 00d6a1a6ffc847a0be2a47c2327ce478 (127.20.88.1:39325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd99b13a460e416b84931323a6dd42eb: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39325: connect: Connection refused (error 111)
I20260928 23:54:58.117918 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25264
I20260928 23:54:58.118803 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.118095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36863 (local address 127.0.0.1:51880)
0928 23:54:58.118624 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.118690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:36863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
I20260928 23:54:58.124302 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 24905
2026-09-28T23:54:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2565 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T23:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:58Z Disabled control of system clock
I20260928 23:54:58.156101 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44803
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:54:58.275465 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:58.275789 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:58.275830 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:58.280184 25440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:58.280313 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:58.280330 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:58.280349 25440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:58.280364 25440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:58.285706 25440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:44803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:58.286936 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:58.288483 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:58.297891 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.297890 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.297952 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:58.298310 25440 server_base.cc:1034] running on GCE node
I20260928 23:54:58.299014 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:58.300367 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:58.301601 25440 hybrid_clock.cc:648] HybridClock initialized: now 1790639698301581 us; error 46 us; skew 500 ppm
I20260928 23:54:58.307818 25440 webserver.cc:460] Webserver started at http://127.20.88.62:41183/ using document root <none> and password file <none>
I20260928 23:54:58.308627 25440 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:58.308744 25440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:58.309028 25440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:58.311426 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "eab04db49c5b4c60bf00b080623296c7"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "3aec1a545eb1abc5d4440c6c74d9f1f1"
server_key_iv: "35757c7cc038e2719102f395b0e8ee30"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.312124 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "eab04db49c5b4c60bf00b080623296c7"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "3aec1a545eb1abc5d4440c6c74d9f1f1"
server_key_iv: "35757c7cc038e2719102f395b0e8ee30"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.318107 25440 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260928 23:54:58.321336 25454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.322954 25440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:54:58.323133 25440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "eab04db49c5b4c60bf00b080623296c7"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "3aec1a545eb1abc5d4440c6c74d9f1f1"
server_key_iv: "35757c7cc038e2719102f395b0e8ee30"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.323282 25440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:58.328451 25440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:58.329185 25440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:58.329406 25440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:58.336812 25440 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:44803
I20260928 23:54:58.336849 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:44803 every 8 connection(s)
I20260928 23:54:58.337874 25440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:58.341372 25507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:58.345541 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25440
I20260928 23:54:58.345682 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:58.345975 20832 external_mini_cluster.cc:1428] Setting key 10c6307e749b81effe6e26465ef3dbdb
I20260928 23:54:58.348402 25507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: Bootstrap starting.
I20260928 23:54:58.350937 25507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:58.351831 25507 log.cc:824] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:58.352653 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.347727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.20.88.62:44803)
0928 23:54:58.348643 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.348662 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.348711 (+ 49us) server_negotiation.cc:407] Connection header received
0928 23:54:58.349452 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.349471 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.349864 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.350266 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.351088 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.351094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.351112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.351125 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.351203 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.351221 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.351349 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.351504 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.351769 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":142,"threads_started":1}
I20260928 23:54:58.354203 25507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: No bootstrap required, opened a new log
I20260928 23:54:58.358284 25507 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } }
I20260928 23:54:58.358573 25507 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:58.358660 25507 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab04db49c5b4c60bf00b080623296c7, State: Initialized, Role: FOLLOWER
I20260928 23:54:58.359172 25507 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } }
I20260928 23:54:58.359335 25507 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:58.359426 25507 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:58.359534 25507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:58.360723 25507 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } }
I20260928 23:54:58.361150 25507 leader_election.cc:302] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab04db49c5b4c60bf00b080623296c7; no voters:
I20260928 23:54:58.361555 25507 leader_election.cc:288] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:58.361723 25512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:58.361985 25512 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [term 1 LEADER]: Becoming Leader. State: Replica: eab04db49c5b4c60bf00b080623296c7, State: Running, Role: LEADER
I20260928 23:54:58.362311 25512 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } }
I20260928 23:54:58.362887 25507 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:58.364105 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab04db49c5b4c60bf00b080623296c7. Latest consensus state: current_term: 1 leader_uuid: "eab04db49c5b4c60bf00b080623296c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } } }
I20260928 23:54:58.364274 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:58.364488 25513 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab04db49c5b4c60bf00b080623296c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab04db49c5b4c60bf00b080623296c7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44803 } } }
I20260928 23:54:58.364660 25513 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7 [sys.catalog]: This master's current role is: LEADER
W20260928 23:54:58.367897 25527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:58.367969 25527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:58.368227 25521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:58.369140 25521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:58.374439 25521 catalog_manager.cc:1347] Generated new cluster ID: 761bb95ed7674c6faad65d5abbb9cc4d
I20260928 23:54:58.374526 25521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:58.387480 25521 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:58.387831 25521 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:58.396049 25521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: Generated new TSK 0
I20260928 23:54:58.396922 25521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:58.408080 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--builtin_ntp_servers=127.20.88.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:54:58.528466 25531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:58.528782 25531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:58.528825 25531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:58.533146 25531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:58.533222 25531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:58.533284 25531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:54:58.538347 25531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:58.539608 25531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:58.541170 25531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:58.551880 25539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:58.551981 25531 server_base.cc:1034] running on GCE node
W20260928 23:54:58.552002 25536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.552131 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:58.552644 25531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:58.553424 25531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:58.554610 25531 hybrid_clock.cc:648] HybridClock initialized: now 1790639698554589 us; error 34 us; skew 500 ppm
I20260928 23:54:58.556910 25531 webserver.cc:460] Webserver started at http://127.20.88.1:43891/ using document root <none> and password file <none>
I20260928 23:54:58.557615 25531 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:58.557687 25531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:58.557927 25531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:58.560312 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "91e58d299f6d48e4a1eca3dee34cf1f5"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "9a74a4d7b339eb014bc399af83081ec1"
server_key_iv: "c892839e17c54e1d2199a8435c196f8c"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.560959 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "91e58d299f6d48e4a1eca3dee34cf1f5"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "9a74a4d7b339eb014bc399af83081ec1"
server_key_iv: "c892839e17c54e1d2199a8435c196f8c"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.565585 25531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 23:54:58.568579 25545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.569973 25531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:54:58.570156 25531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "91e58d299f6d48e4a1eca3dee34cf1f5"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "9a74a4d7b339eb014bc399af83081ec1"
server_key_iv: "c892839e17c54e1d2199a8435c196f8c"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.570299 25531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:58.575345 25531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:58.576050 25531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:58.576316 25531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:58.576920 25531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:58.577943 25531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:58.578018 25531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.578086 25531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:58.578146 25531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.588058 25531 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:34707
I20260928 23:54:58.588095 25658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:34707 every 8 connection(s)
I20260928 23:54:58.589138 25531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:54:58.595391 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25531
I20260928 23:54:58.595570 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:54:58.596077 20832 external_mini_cluster.cc:1428] Setting key b05e8efd9913c12b61e9b385a92234eb
I20260928 23:54:58.596464 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.591178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:42343 (local address 127.20.88.62:44803)
0928 23:54:58.591341 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.591346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.592687 (+ 1341us) server_negotiation.cc:407] Connection header received
0928 23:54:58.594094 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.594097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.594143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.594230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.595700 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.595706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.595710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.595730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.595764 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.595777 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.595857 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.596141 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.596265 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:54:58.597841 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.591707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44803 (local address 127.20.88.1:42343)
0928 23:54:58.592525 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.592571 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.593835 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.594414 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.594422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.594527 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.595228 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.595230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.595284 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.595435 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.595940 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.595952 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.595959 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.596190 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.596669 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260928 23:54:58.598563 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--builtin_ntp_servers=127.20.88.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:54:58.599213 25659 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44803
I20260928 23:54:58.599494 25659 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:58.600085 25659 heartbeater.cc:511] Master 127.20.88.62:44803 requested a full tablet report, sending...
I20260928 23:54:58.601981 25471 ts_manager.cc:194] Registered new tserver with Master: 91e58d299f6d48e4a1eca3dee34cf1f5 (127.20.88.1:34707)
W20260928 23:54:58.714534 25662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:58.714913 25662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:58.714984 25662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:58.719153 25662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:58.719231 25662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:58.719292 25662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:54:58.724480 25662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:58.725708 25662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:58.727233 25662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:58.737465 25670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.737473 25667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.737481 25668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:58.737646 25662 server_base.cc:1034] running on GCE node
I20260928 23:54:58.738392 25662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:58.739068 25662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:58.740330 25662 hybrid_clock.cc:648] HybridClock initialized: now 1790639698740298 us; error 42 us; skew 500 ppm
I20260928 23:54:58.742321 25662 webserver.cc:460] Webserver started at http://127.20.88.2:33413/ using document root <none> and password file <none>
I20260928 23:54:58.743008 25662 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:58.743132 25662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:58.743377 25662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:58.745623 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "a8d49cbd4b6f4e4a9526dd57abc44736"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "716aabdb9f552af803fe12c9018543b6"
server_key_iv: "2a8fc8b37dfcb21fd51da07a9573e6e6"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.746227 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "a8d49cbd4b6f4e4a9526dd57abc44736"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "716aabdb9f552af803fe12c9018543b6"
server_key_iv: "2a8fc8b37dfcb21fd51da07a9573e6e6"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.750494 25662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:58.753221 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.754544 25662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:58.754693 25662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "a8d49cbd4b6f4e4a9526dd57abc44736"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "716aabdb9f552af803fe12c9018543b6"
server_key_iv: "2a8fc8b37dfcb21fd51da07a9573e6e6"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.754822 25662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:58.759850 25662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:58.760581 25662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:58.760771 25662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:58.761377 25662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:58.762367 25662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:58.762439 25662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.762516 25662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:58.762567 25662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.771999 25662 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:40741
I20260928 23:54:58.772040 25789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:40741 every 8 connection(s)
I20260928 23:54:58.773082 25662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:54:58.775873 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25662
I20260928 23:54:58.776008 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:54:58.776378 20832 external_mini_cluster.cc:1428] Setting key 5b4081f1b57f00d229d438e32baf699c
I20260928 23:54:58.778419 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--builtin_ntp_servers=127.20.88.20:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:54:58.780368 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.775211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:52763 (local address 127.20.88.62:44803)
0928 23:54:58.775372 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.775378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.776869 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 23:54:58.778416 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.778419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.778462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.778541 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.779844 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.779848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.779851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.779864 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.779890 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.779904 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.780003 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.780164 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.780213 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:54:58.781550 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.775774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44803 (local address 127.20.88.2:52763)
0928 23:54:58.776681 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.776724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.778192 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.778735 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.778743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.778849 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.779536 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.779540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.779600 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.779684 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.779987 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.779995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.780000 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.780206 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.780490 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":153,"threads_started":1}
I20260928 23:54:58.782864 25790 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44803
I20260928 23:54:58.783152 25790 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:58.783602 25790 heartbeater.cc:511] Master 127.20.88.62:44803 requested a full tablet report, sending...
I20260928 23:54:58.784802 25471 ts_manager.cc:194] Registered new tserver with Master: a8d49cbd4b6f4e4a9526dd57abc44736 (127.20.88.2:40741)
W20260928 23:54:58.897964 25793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:58.898289 25793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:58.898375 25793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:58.902696 25793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:58.902809 25793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:58.902905 25793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:54:58.908128 25793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:58.909451 25793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:58.910964 25793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:58.921639 25798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.921639 25799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:58.921639 25801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:58.921686 25793 server_base.cc:1034] running on GCE node
I20260928 23:54:58.922379 25793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:58.923038 25793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:58.924202 25793 hybrid_clock.cc:648] HybridClock initialized: now 1790639698924182 us; error 30 us; skew 500 ppm
I20260928 23:54:58.926206 25793 webserver.cc:460] Webserver started at http://127.20.88.3:46161/ using document root <none> and password file <none>
I20260928 23:54:58.926823 25793 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:58.926885 25793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:58.927139 25793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:58.929464 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "fbad29e5da9d4912981f1fbd620c7fc2"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "02b344b43ad58e28c1ed0fd87eb85b94"
server_key_iv: "7565180978c9c1cd07935f2562bc6b17"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.930110 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "fbad29e5da9d4912981f1fbd620c7fc2"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "02b344b43ad58e28c1ed0fd87eb85b94"
server_key_iv: "7565180978c9c1cd07935f2562bc6b17"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.934109 25793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:54:58.936925 25807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.938205 25793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:58.938361 25793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "fbad29e5da9d4912981f1fbd620c7fc2"
format_stamp: "Formatted at 2026-09-28 23:54:58 on dist-test-slave-wvpw"
server_key: "02b344b43ad58e28c1ed0fd87eb85b94"
server_key_iv: "7565180978c9c1cd07935f2562bc6b17"
server_key_version: "encryptionkey@0"
I20260928 23:54:58.938493 25793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:58.943778 25793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:58.944542 25793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:58.944746 25793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:58.945324 25793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:54:58.946293 25793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:54:58.946365 25793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.946439 25793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:54:58.946489 25793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:58.955622 25793 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:37573
I20260928 23:54:58.955751 25920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:37573 every 8 connection(s)
I20260928 23:54:58.956748 25793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:54:58.963750 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.958778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:40853 (local address 127.20.88.62:44803)
0928 23:54:58.958990 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.958996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.960323 (+ 1327us) server_negotiation.cc:407] Connection header received
0928 23:54:58.961670 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.961674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.961728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.961811 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.963131 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.963134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.963136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.963149 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.963171 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.963179 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.963317 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.963506 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.963555 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 23:54:58.964708 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.959308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44803 (local address 127.20.88.3:40853)
0928 23:54:58.960082 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.960126 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.961436 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.962039 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.962047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.962144 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.962833 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.962834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.962879 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.962953 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.963302 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.963308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.963313 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.963497 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.963750 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":187,"threads_started":1}
I20260928 23:54:58.965904 25921 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44803
I20260928 23:54:58.966060 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25793
I20260928 23:54:58.966166 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:54:58.966166 25921 heartbeater.cc:464] Registering TS with master...
I20260928 23:54:58.966465 20832 external_mini_cluster.cc:1428] Setting key 28996e9e10ffa402ebc725f2549271be
I20260928 23:54:58.966686 25921 heartbeater.cc:511] Master 127.20.88.62:44803 requested a full tablet report, sending...
I20260928 23:54:58.967904 25471 ts_manager.cc:194] Registered new tserver with Master: fbad29e5da9d4912981f1fbd620c7fc2 (127.20.88.3:37573)
I20260928 23:54:58.968902 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:54:58.972623 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.970877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.20.88.62:44803)
0928 23:54:58.971037 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.971042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.971203 (+ 161us) server_negotiation.cc:407] Connection header received
0928 23:54:58.971368 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.971371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.971423 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.971507 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.972207 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.972210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.972213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.972225 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.972272 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.972282 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.972378 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.972449 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.972490 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:54:58.975956 25470 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:54:58.979156 25470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:54:58.992566 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.989769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.20.88.1:34707)
0928 23:54:58.990172 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.990185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.990211 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:54:58.990267 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.990273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.990491 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.991071 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.992068 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.992073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.992080 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.992091 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.992123 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.992146 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.992268 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.992324 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.992404 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260928 23:54:58.992555 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.988996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:34707 (local address 127.0.0.1:44272)
0928 23:54:58.989442 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.989474 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.989587 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.990663 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.990669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.990753 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.991432 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.991436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.991461 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.991556 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.992219 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.992223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.992250 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.992309 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.992358 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"spinlock_wait_cycles":6400,"thread_start_us":122,"threads_started":1}
I20260928 23:54:58.992614 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.989049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.20.88.2:40741)
0928 23:54:58.990288 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.990299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.990318 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:54:58.990363 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.990367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.990490 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.991241 (+ 751us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.991895 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.991904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.991922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.991938 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.991989 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.992019 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.992151 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.992396 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.992469 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":135,"threads_started":1}
I20260928 23:54:58.992561 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.988909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40741 (local address 127.0.0.1:55760)
0928 23:54:58.989414 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.989467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.989586 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.990663 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.990669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.990753 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.991713 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.991717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.991730 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.991761 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.992117 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.992127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.992132 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.992266 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.992435 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":86,"spinlock_wait_cycles":7936,"thread_start_us":95,"threads_started":1}
I20260928 23:54:58.993830 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.989913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.20.88.3:37573)
0928 23:54:58.990340 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:58.990351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:58.990657 (+ 306us) server_negotiation.cc:407] Connection header received
0928 23:54:58.990724 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:58.990728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:58.990864 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:58.991014 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:58.991987 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:58.991993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:58.992004 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:58.992017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:58.992062 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:58.992096 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:58.992206 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:58.993646 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0928 23:54:58.993709 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
I20260928 23:54:58.994302 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:58.989688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:37573 (local address 127.0.0.1:32982)
0928 23:54:58.990481 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:58.990496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:58.990566 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:58.990992 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:58.990995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:58.991009 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:58.991800 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:58.991801 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:58.991832 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:58.991863 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:58.992275 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:58.992281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:58.992288 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:54:58.992360 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:54:58.992396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":102,"threads_started":1}
I20260928 23:54:58.995940 25593 tablet_service.cc:1467] Processing CreateTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c (DEFAULT_TABLE table=test-table [id=2c86b498bc6f4c23ad89a95313bc72d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:58.995975 25724 tablet_service.cc:1467] Processing CreateTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c (DEFAULT_TABLE table=test-table [id=2c86b498bc6f4c23ad89a95313bc72d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:58.997129 25724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed981c0d5dab4361b8eaed4a14837f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:58.997131 25593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed981c0d5dab4361b8eaed4a14837f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:58.997236 25855 tablet_service.cc:1467] Processing CreateTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c (DEFAULT_TABLE table=test-table [id=2c86b498bc6f4c23ad89a95313bc72d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:54:58.998298 25855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed981c0d5dab4361b8eaed4a14837f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:59.003944 25938 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Bootstrap starting.
I20260928 23:54:59.005625 25939 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Bootstrap starting.
I20260928 23:54:59.006718 25938 tablet_bootstrap.cc:654] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:59.007675 25938 log.cc:824] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:59.008003 25939 tablet_bootstrap.cc:654] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:59.008544 25940 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Bootstrap starting.
I20260928 23:54:59.008927 25939 log.cc:824] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:59.010001 25938 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: No bootstrap required, opened a new log
I20260928 23:54:59.010214 25938 ts_tablet_manager.cc:1397] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:59.010804 25940 tablet_bootstrap.cc:654] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:59.011080 25939 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: No bootstrap required, opened a new log
I20260928 23:54:59.011286 25939 ts_tablet_manager.cc:1397] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:59.011623 25940 log.cc:824] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:59.013634 25940 tablet_bootstrap.cc:492] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: No bootstrap required, opened a new log
I20260928 23:54:59.013847 25940 ts_tablet_manager.cc:1397] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:54:59.015204 25938 raft_consensus.cc:348] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.015528 25938 raft_consensus.cc:374] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:59.015594 25938 raft_consensus.cc:729] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e58d299f6d48e4a1eca3dee34cf1f5, State: Initialized, Role: FOLLOWER
I20260928 23:54:59.015872 25939 raft_consensus.cc:348] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.016142 25939 raft_consensus.cc:374] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:59.016088 25938 consensus_queue.cc:260] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.016197 25939 raft_consensus.cc:729] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d49cbd4b6f4e4a9526dd57abc44736, State: Initialized, Role: FOLLOWER
I20260928 23:54:59.016654 25939 consensus_queue.cc:260] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.017407 25790 heartbeater.cc:503] Master 127.20.88.62:44803 was elected leader, sending a full tablet report...
I20260928 23:54:59.017433 25938 ts_tablet_manager.cc:1428] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260928 23:54:59.017885 25659 heartbeater.cc:503] Master 127.20.88.62:44803 was elected leader, sending a full tablet report...
I20260928 23:54:59.018543 25939 ts_tablet_manager.cc:1428] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 23:54:59.019184 25940 raft_consensus.cc:348] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.019486 25940 raft_consensus.cc:374] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:59.019554 25940 raft_consensus.cc:729] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbad29e5da9d4912981f1fbd620c7fc2, State: Initialized, Role: FOLLOWER
I20260928 23:54:59.020015 25940 consensus_queue.cc:260] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.020812 25921 heartbeater.cc:503] Master 127.20.88.62:44803 was elected leader, sending a full tablet report...
I20260928 23:54:59.021806 25940 ts_tablet_manager.cc:1428] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 23:54:59.024097 25791 tablet.cc:2369] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:54:59.090423 25660 tablet.cc:2369] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:59.187947 25944 raft_consensus.cc:482] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:54:59.188305 25944 raft_consensus.cc:504] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.189585 25944 leader_election.cc:288] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d49cbd4b6f4e4a9526dd57abc44736 (127.20.88.2:40741), fbad29e5da9d4912981f1fbd620c7fc2 (127.20.88.3:37573)
I20260928 23:54:59.192040 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.189986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40741 (local address 127.20.88.1:46215)
0928 23:54:59.190295 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.190311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:59.190461 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:59.190743 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:59.190746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:59.190760 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:59.191340 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:59.191343 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:59.191357 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:59.191396 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:59.191717 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:59.191725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:59.191728 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:54:59.191814 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:54:59.191855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
I20260928 23:54:59.192041 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.190083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:46215 (local address 127.20.88.2:40741)
0928 23:54:59.190265 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.190270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.190412 (+ 142us) server_negotiation.cc:407] Connection header received
0928 23:54:59.190579 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.190583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.190627 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.190713 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:59.191552 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:59.191556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:59.191559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:59.191572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:59.191592 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:59.191602 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:59.191718 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:59.191830 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 23:54:59.191870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 23:54:59.192216 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.190504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:52681 (local address 127.20.88.3:37573)
0928 23:54:59.190681 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.190688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.190730 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:54:59.190865 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.190867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.190898 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.191010 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:59.191780 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:59.191786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:59.191792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:59.191806 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:59.191830 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:59.191841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:59.191925 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:59.192033 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:54:59.192078 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:54:59.192619 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.190380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:37573 (local address 127.20.88.1:52681)
0928 23:54:59.190659 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.190672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:59.190755 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:59.191020 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:59.191023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:59.191041 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:59.191543 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:54:59.191545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:59.191555 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:54:59.191638 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:59.191929 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:59.191933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:59.191936 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:54:59.192019 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:54:59.192057 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260928 23:54:59.192715 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed981c0d5dab4361b8eaed4a14837f8c" candidate_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" is_pre_election: true
I20260928 23:54:59.193069 25744 raft_consensus.cc:2454] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e58d299f6d48e4a1eca3dee34cf1f5 in term 0.
I20260928 23:54:59.193331 25875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed981c0d5dab4361b8eaed4a14837f8c" candidate_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" is_pre_election: true
I20260928 23:54:59.193709 25875 raft_consensus.cc:2454] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e58d299f6d48e4a1eca3dee34cf1f5 in term 0.
I20260928 23:54:59.193732 25547 leader_election.cc:302] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e58d299f6d48e4a1eca3dee34cf1f5, a8d49cbd4b6f4e4a9526dd57abc44736; no voters:
I20260928 23:54:59.194089 25944 raft_consensus.cc:2781] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:54:59.194167 25944 raft_consensus.cc:482] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:54:59.194231 25944 raft_consensus.cc:3037] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:59.195322 25944 raft_consensus.cc:504] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.195703 25944 leader_election.cc:288] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [CANDIDATE]: Term 1 election: Requested vote from peers a8d49cbd4b6f4e4a9526dd57abc44736 (127.20.88.2:40741), fbad29e5da9d4912981f1fbd620c7fc2 (127.20.88.3:37573)
I20260928 23:54:59.196206 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed981c0d5dab4361b8eaed4a14837f8c" candidate_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736"
I20260928 23:54:59.196321 25875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed981c0d5dab4361b8eaed4a14837f8c" candidate_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbad29e5da9d4912981f1fbd620c7fc2"
I20260928 23:54:59.196434 25875 raft_consensus.cc:3037] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:59.196466 25744 raft_consensus.cc:3037] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:59.197542 25875 raft_consensus.cc:2454] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e58d299f6d48e4a1eca3dee34cf1f5 in term 1.
I20260928 23:54:59.197654 25744 raft_consensus.cc:2454] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e58d299f6d48e4a1eca3dee34cf1f5 in term 1.
I20260928 23:54:59.197922 25547 leader_election.cc:302] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e58d299f6d48e4a1eca3dee34cf1f5, fbad29e5da9d4912981f1fbd620c7fc2; no voters:
I20260928 23:54:59.198124 25944 raft_consensus.cc:2781] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:59.198342 25944 raft_consensus.cc:686] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [term 1 LEADER]: Becoming Leader. State: Replica: 91e58d299f6d48e4a1eca3dee34cf1f5, State: Running, Role: LEADER
I20260928 23:54:59.198750 25944 consensus_queue.cc:237] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } }
I20260928 23:54:59.202253 25469 catalog_manager.cc:5495] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91e58d299f6d48e4a1eca3dee34cf1f5 (127.20.88.1). New cstate: current_term: 1 leader_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91e58d299f6d48e4a1eca3dee34cf1f5" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 } health_report { overall_health: UNKNOWN } } }
W20260928 23:54:59.207762 25922 tablet.cc:2369] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:59.231781 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.230350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.20.88.1:34707)
0928 23:54:59.230474 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.230478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.230490 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:59.230571 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.230573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.230639 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.230711 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:59.231407 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:59.231410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:59.231412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:59.231424 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:59.231445 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:59.231454 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:59.231531 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:59.231633 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:54:59.231670 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:54:59.238667 25744 raft_consensus.cc:1261] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 1 FOLLOWER]: Refusing update from remote peer 91e58d299f6d48e4a1eca3dee34cf1f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:59.238667 25875 raft_consensus.cc:1261] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 1 FOLLOWER]: Refusing update from remote peer 91e58d299f6d48e4a1eca3dee34cf1f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:59.239606 25949 consensus_queue.cc:1035] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d49cbd4b6f4e4a9526dd57abc44736" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:59.239838 25944 consensus_queue.cc:1035] T ed981c0d5dab4361b8eaed4a14837f8c P 91e58d299f6d48e4a1eca3dee34cf1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbad29e5da9d4912981f1fbd620c7fc2" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:59.250476 25953 mvcc.cc:205] Tried to move back new op lower bound from 7334460208071950336 to 7334460207921053696. Current Snapshot: MvccSnapshot[applied={T|T < 7334460208071950336}]
I20260928 23:54:59.250701 25954 mvcc.cc:205] Tried to move back new op lower bound from 7334460208071950336 to 7334460207921053696. Current Snapshot: MvccSnapshot[applied={T|T < 7334460208071950336}]
I20260928 23:54:59.258639 25469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:59.258888 25469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:54:59.262979 25469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eab04db49c5b4c60bf00b080623296c7: Sending DeleteTablet for 3 replicas of tablet ed981c0d5dab4361b8eaed4a14837f8c
I20260928 23:54:59.264405 25724 tablet_service.cc:1514] Processing DeleteTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:59 UTC) from {username='slave'} at 127.0.0.1:55760
I20260928 23:54:59.264570 25855 tablet_service.cc:1514] Processing DeleteTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:59 UTC) from {username='slave'} at 127.0.0.1:32982
I20260928 23:54:59.265482 25962 tablet_replica.cc:330] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: stopping tablet replica
I20260928 23:54:59.265532 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25531
I20260928 23:54:59.265553 25593 tablet_service.cc:1514] Processing DeleteTablet for tablet ed981c0d5dab4361b8eaed4a14837f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:54:59 UTC) from {username='slave'} at 127.0.0.1:44272
I20260928 23:54:59.265970 25962 raft_consensus.cc:2229] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:59.266461 25962 raft_consensus.cc:2258] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:59.268481 25963 tablet_replica.cc:330] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: stopping tablet replica
I20260928 23:54:59.268596 25962 ts_tablet_manager.cc:1905] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:59.269125 25963 raft_consensus.cc:2229] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:54:59.269671 25963 raft_consensus.cc:2258] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:54:59.271837 25963 ts_tablet_manager.cc:1905] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:54:59.273909 25962 ts_tablet_manager.cc:1918] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:59.274055 25962 log.cc:1196] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/ed981c0d5dab4361b8eaed4a14837f8c
I20260928 23:54:59.274399 25962 ts_tablet_manager.cc:1939] T ed981c0d5dab4361b8eaed4a14837f8c P fbad29e5da9d4912981f1fbd620c7fc2: Deleting consensus metadata
I20260928 23:54:59.274816 25963 ts_tablet_manager.cc:1918] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:54:59.274922 25963 log.cc:1196] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/ed981c0d5dab4361b8eaed4a14837f8c
I20260928 23:54:59.275254 25963 ts_tablet_manager.cc:1939] T ed981c0d5dab4361b8eaed4a14837f8c P a8d49cbd4b6f4e4a9526dd57abc44736: Deleting consensus metadata
W20260928 23:54:59.275535 25458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:34707 (error 108)
I20260928 23:54:59.275933 25456 catalog_manager.cc:4841] TS fbad29e5da9d4912981f1fbd620c7fc2 (127.20.88.3:37573): tablet ed981c0d5dab4361b8eaed4a14837f8c (table test-table [id=2c86b498bc6f4c23ad89a95313bc72d3]) successfully deleted
I20260928 23:54:59.276494 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25662
I20260928 23:54:59.277333 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.276630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:34707 (local address 127.0.0.1:44294)
0928 23:54:59.277160 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.277253 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
W20260928 23:54:59.277493 25458 catalog_manager.cc:4584] TS 91e58d299f6d48e4a1eca3dee34cf1f5 (127.20.88.1:34707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed981c0d5dab4361b8eaed4a14837f8c: Network error: Client connection negotiation failed: client connection to 127.20.88.1:34707: connect: Connection refused (error 111)
I20260928 23:54:59.283129 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25793
I20260928 23:54:59.284994 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.283299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40741 (local address 127.0.0.1:55770)
0928 23:54:59.284795 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.284872 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1371}
I20260928 23:54:59.289812 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25440
2026-09-28T23:54:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1164 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 23:54:59.312454 20832 mini_oidc.cc:152] Starting JWKS server
I20260928 23:54:59.314105 20832 webserver.cc:460] Webserver started at https://127.0.0.1:43145/ using document root <none> and password file <none>
I20260928 23:54:59.314306 20832 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:54:59.314603 20832 webserver.cc:460] Webserver started at http://127.0.0.1:33229/ using document root <none> and password file <none>
2026-09-28T23:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:59Z Disabled control of system clock
I20260928 23:54:59.325408 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44489
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:44489
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/testchainca.cert with env {}
W20260928 23:54:59.443528 25970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:59.443847 25970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:59.443893 25970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:59.448190 25970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:59.448365 25970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:59.448386 25970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:59.448406 25970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:59.448423 25970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:54:59.448441 25970 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:54:59.448467 25970 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43145/jwks.json
I20260928 23:54:59.454548 25970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:44489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:59.455757 25970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:59.456934 25970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:59.466327 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:59.466327 25978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:59.466358 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:59.466476 25970 server_base.cc:1034] running on GCE node
I20260928 23:54:59.467428 25970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:59.468748 25970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:59.469950 25970 hybrid_clock.cc:648] HybridClock initialized: now 1790639699469913 us; error 52 us; skew 500 ppm
I20260928 23:54:59.472056 25970 webserver.cc:460] Webserver started at http://127.20.88.62:33067/ using document root <none> and password file <none>
I20260928 23:54:59.472774 25970 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:59.472875 25970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:59.473119 25970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:59.475337 25970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "f87834f5cebf4b5ba8e6fe097c062291"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "c7312803d97d66aa89d32fb69135927d"
server_key_iv: "99af5036855e1695724f9810f44991d1"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.475952 25970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "f87834f5cebf4b5ba8e6fe097c062291"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "c7312803d97d66aa89d32fb69135927d"
server_key_iv: "99af5036855e1695724f9810f44991d1"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.479935 25970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:54:59.482796 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:59.484196 25970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:54:59.484424 25970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "f87834f5cebf4b5ba8e6fe097c062291"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "c7312803d97d66aa89d32fb69135927d"
server_key_iv: "99af5036855e1695724f9810f44991d1"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.484563 25970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:59.534461 25970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:59.535283 25970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:59.535516 25970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:59.543187 25970 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:44489
I20260928 23:54:59.543195 26036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:44489 every 8 connection(s)
I20260928 23:54:59.544310 25970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:59.547773 26037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:59.552990 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 25970
I20260928 23:54:59.553121 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:59.553426 20832 external_mini_cluster.cc:1428] Setting key ed1b0229f3574c80a3f9059cbb1fb857
I20260928 23:54:59.555063 26037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291: Bootstrap starting.
I20260928 23:54:59.557694 26037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:59.558650 26037 log.cc:824] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:59.560995 26037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291: No bootstrap required, opened a new log
I20260928 23:54:59.563758 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.554982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.20.88.62:44489)
0928 23:54:59.555756 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.555775 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.555814 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:54:59.556541 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.556560 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.557005 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.557402 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:59.558037 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.559183 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.560158 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.560693 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.560876 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.561817 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:59.561823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:59.561838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:59.562305 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:59.562366 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:59.562388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:59.562498 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:59.562686 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:54:59.562969 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260928 23:54:59.565063 26037 raft_consensus.cc:348] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } }
I20260928 23:54:59.565375 26037 raft_consensus.cc:374] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:59.565424 26037 raft_consensus.cc:729] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87834f5cebf4b5ba8e6fe097c062291, State: Initialized, Role: FOLLOWER
I20260928 23:54:59.565945 26037 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } }
I20260928 23:54:59.566159 26037 raft_consensus.cc:388] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:59.566249 26037 raft_consensus.cc:482] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:59.566360 26037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:59.567159 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:59.567813 26037 raft_consensus.cc:504] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } }
I20260928 23:54:59.568286 26037 leader_election.cc:302] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f87834f5cebf4b5ba8e6fe097c062291; no voters:
I20260928 23:54:59.568733 26037 leader_election.cc:288] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:59.568813 26042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:59.569016 26042 raft_consensus.cc:686] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [term 1 LEADER]: Becoming Leader. State: Replica: f87834f5cebf4b5ba8e6fe097c062291, State: Running, Role: LEADER
I20260928 23:54:59.569355 26042 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } }
I20260928 23:54:59.570168 26037 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:59.571291 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87834f5cebf4b5ba8e6fe097c062291. Latest consensus state: current_term: 1 leader_uuid: "f87834f5cebf4b5ba8e6fe097c062291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } } }
I20260928 23:54:59.571321 26043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f87834f5cebf4b5ba8e6fe097c062291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87834f5cebf4b5ba8e6fe097c062291" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44489 } } }
I20260928 23:54:59.571448 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:59.571462 26043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:59.571863 26051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:59.575598 26051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:59.582152 26051 catalog_manager.cc:1347] Generated new cluster ID: e54f701c857746b69a7dc711588337e4
I20260928 23:54:59.582247 26051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:59.604921 26051 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:59.605943 26051 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:59.610971 26051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f87834f5cebf4b5ba8e6fe097c062291: Generated new TSK 0
I20260928 23:54:59.611830 26051 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:54:59.645716 26067 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 23:54:59.645850 26067 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:54:59.646334 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.621614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.20.88.62:44489)
0928 23:54:59.621784 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.621789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.622035 (+ 246us) server_negotiation.cc:407] Connection header received
0928 23:54:59.622292 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.622295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.622346 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.622427 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:54:59.622725 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.623277 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.624537 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.624776 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.624861 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.625286 (+ 425us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:54:59.645898 (+ 20612us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:54:59.646211 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 23:54:59.646453 26040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 23:54:59.647876 26066 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:59.621541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44489 (local address 127.0.0.1:47164)
0928 23:54:59.621938 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.621955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:59.622094 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:59.622468 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:59.622471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:59.622479 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:54:59.622617 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:59.622624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.623393 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.623397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:59.624415 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:59.624427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.624902 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.624906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:59.625016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.625083 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:54:59.646191 (+ 21108us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:54:59.646463 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.88.62:44489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.88.62:44489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 23:54:59.648892 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 25970
2026-09-28T23:54:59Z chronyd exiting
I20260928 23:54:59.687165 20832 test_util.cc:175] -----------------------------------------------
I20260928 23:54:59.687268 20832 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639669102918-20832-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (376 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 23:54:59.688393 20832 mini_oidc.cc:152] Starting JWKS server
I20260928 23:54:59.689304 20832 webserver.cc:460] Webserver started at https://127.0.0.1:36541/ using document root <none> and password file <none>
I20260928 23:54:59.689425 20832 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:54:59.689715 20832 webserver.cc:460] Webserver started at http://127.0.0.1:39299/ using document root <none> and password file <none>
2026-09-28T23:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:59Z Disabled control of system clock
I20260928 23:54:59.699445 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36327
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:36327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36541/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/testchainca.cert with env {}
W20260928 23:54:59.821156 26073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:59.821477 26073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:59.821520 26073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:59.825783 26073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:59.825855 26073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:59.825870 26073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:59.825888 26073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:59.825904 26073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:54:59.825922 26073 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:54:59.825934 26073 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36541/jwks.json
I20260928 23:54:59.831844 26073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:36327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36541/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:59.833093 26073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:59.834136 26073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:59.843724 26078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:59.843813 26079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:59.843724 26081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:59.843997 26073 server_base.cc:1034] running on GCE node
I20260928 23:54:59.844764 26073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:59.846028 26073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:59.847242 26073 hybrid_clock.cc:648] HybridClock initialized: now 1790639699847234 us; error 40 us; skew 500 ppm
I20260928 23:54:59.849445 26073 webserver.cc:460] Webserver started at http://127.20.88.62:39157/ using document root <none> and password file <none>
I20260928 23:54:59.850087 26073 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:59.850186 26073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:59.850421 26073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:59.852864 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "bff880eb552048da977bba5ba458884d"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "470fa7a6499a92fc38e6ea32669ec1b6"
server_key_iv: "5b92aaf14aeb8cc137c300b0b82fd48e"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.853475 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "bff880eb552048da977bba5ba458884d"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "470fa7a6499a92fc38e6ea32669ec1b6"
server_key_iv: "5b92aaf14aeb8cc137c300b0b82fd48e"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.857597 26073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:54:59.860697 26087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:59.862061 26073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:54:59.862233 26073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "bff880eb552048da977bba5ba458884d"
format_stamp: "Formatted at 2026-09-28 23:54:59 on dist-test-slave-wvpw"
server_key: "470fa7a6499a92fc38e6ea32669ec1b6"
server_key_iv: "5b92aaf14aeb8cc137c300b0b82fd48e"
server_key_version: "encryptionkey@0"
I20260928 23:54:59.862380 26073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:59.897289 26073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:59.898108 26073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:59.898331 26073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:59.905668 26073 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:36327
I20260928 23:54:59.905689 26139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:36327 every 8 connection(s)
I20260928 23:54:59.906705 26073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:59.906940 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26073
I20260928 23:54:59.907071 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:54:59.907401 20832 external_mini_cluster.cc:1428] Setting key 6d258d8c63b0b8d612ccc0184cb4eb9c
I20260928 23:54:59.910450 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:59.917241 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.908905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.20.88.62:36327)
0928 23:54:59.909557 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.909576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.909613 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:54:59.910272 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.910289 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.910750 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.911053 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:59.911396 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.912476 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.913701 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.914240 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.914360 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.915331 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:59.915336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:59.915349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:59.915753 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:59.915809 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:59.915829 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:59.915907 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:59.916129 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 23:54:59.916483 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":143,"threads_started":1}
I20260928 23:54:59.917495 26140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d: Bootstrap starting.
I20260928 23:54:59.920055 26140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:59.920463 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:59.920930 26140 log.cc:824] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:59.923095 26140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d: No bootstrap required, opened a new log
I20260928 23:54:59.926896 26140 raft_consensus.cc:348] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } }
I20260928 23:54:59.927213 26140 raft_consensus.cc:374] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:59.927274 26140 raft_consensus.cc:729] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff880eb552048da977bba5ba458884d, State: Initialized, Role: FOLLOWER
I20260928 23:54:59.927742 26140 consensus_queue.cc:260] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } }
I20260928 23:54:59.927877 26140 raft_consensus.cc:388] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:59.927920 26140 raft_consensus.cc:482] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:59.927989 26140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:59.929080 26140 raft_consensus.cc:504] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } }
I20260928 23:54:59.929415 26140 leader_election.cc:302] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bff880eb552048da977bba5ba458884d; no voters:
I20260928 23:54:59.929744 26140 leader_election.cc:288] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:59.929879 26145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:59.930156 26145 raft_consensus.cc:686] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [term 1 LEADER]: Becoming Leader. State: Replica: bff880eb552048da977bba5ba458884d, State: Running, Role: LEADER
I20260928 23:54:59.930519 26145 consensus_queue.cc:237] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } }
I20260928 23:54:59.931133 26140 sys_catalog.cc:564] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:59.932410 26147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bff880eb552048da977bba5ba458884d. Latest consensus state: current_term: 1 leader_uuid: "bff880eb552048da977bba5ba458884d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } } }
I20260928 23:54:59.932449 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bff880eb552048da977bba5ba458884d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff880eb552048da977bba5ba458884d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36327 } } }
I20260928 23:54:59.932557 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:59.932556 26147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:59.932998 26154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:59.936389 26154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:59.943262 26154 catalog_manager.cc:1347] Generated new cluster ID: ac49d3a0e3bf4ed790d604f8edd3afbb
I20260928 23:54:59.943348 26154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:59.968568 26154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:59.969929 26154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:59.976529 26154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bff880eb552048da977bba5ba458884d: Generated new TSK 0
I20260928 23:54:59.977469 26154 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:55:00.018303 26170 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 23:55:00.018414 26170 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:55:00.019132 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:59.995263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.20.88.62:36327)
0928 23:54:59.995436 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:59.995442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:59.995557 (+ 115us) server_negotiation.cc:407] Connection header received
0928 23:54:59.995731 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:59.995733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:59.995785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:59.995885 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:54:59.996182 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.996854 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.998177 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.998445 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.998555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.998934 (+ 379us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:55:00.018713 (+ 19779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36541/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:55:00.018980 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36541/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 23:55:00.019236 26143 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36541/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 23:55:00.019356 26169 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:59.995103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36327 (local address 127.0.0.1:56894)
0928 23:54:59.995453 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:59.995469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:59.995591 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:59.995906 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:59.995909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:59.995917 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:54:59.996039 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:59.996046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.996983 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.996987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:59.998022 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:59.998035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:59.998588 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:59.998592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:59.998692 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:59.998724 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:55:00.018949 (+ 20225us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36541/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:55:00.019170 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.88.62:36327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36541/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260928 23:55:00.019811 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26073
2026-09-28T23:55:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 23:55:00.059929 20832 mini_oidc.cc:152] Starting JWKS server
I20260928 23:55:00.060979 20832 webserver.cc:460] Webserver started at https://127.0.0.1:44005/ using document root <none> and password file <none>
I20260928 23:55:00.061084 20832 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:55:00.061302 20832 webserver.cc:460] Webserver started at http://127.0.0.1:37345/ using document root <none> and password file <none>
2026-09-28T23:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:00Z Disabled control of system clock
I20260928 23:55:00.071122 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32937
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:32937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44005/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:55:00.194012 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:00.194350 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:00.194391 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:00.198623 26176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:00.198701 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:00.198719 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:00.198736 26176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:00.198752 26176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:55:00.198769 26176 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:55:00.198783 26176 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44005/jwks.json
I20260928 23:55:00.203950 26176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:32937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44005/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:00.205219 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:00.206586 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:00.215687 26181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:00.215906 26176 server_base.cc:1034] running on GCE node
W20260928 23:55:00.215675 26182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:00.215677 26184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:00.216711 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:00.217908 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:00.219137 26176 hybrid_clock.cc:648] HybridClock initialized: now 1790639700219102 us; error 53 us; skew 500 ppm
I20260928 23:55:00.221371 26176 webserver.cc:460] Webserver started at http://127.20.88.62:35939/ using document root <none> and password file <none>
I20260928 23:55:00.222030 26176 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:00.222105 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:00.222335 26176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:00.224679 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "c642507377314374ab8675823ba62aea"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "a0db517e6e44cb2ced89081bd6421c0e"
server_key_iv: "27b928de38a7941292b574be2c8fb4b3"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.225306 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "c642507377314374ab8675823ba62aea"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "a0db517e6e44cb2ced89081bd6421c0e"
server_key_iv: "27b928de38a7941292b574be2c8fb4b3"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.229576 26176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:00.232349 26190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:00.233724 26176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:00.233881 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "c642507377314374ab8675823ba62aea"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "a0db517e6e44cb2ced89081bd6421c0e"
server_key_iv: "27b928de38a7941292b574be2c8fb4b3"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.234014 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:00.248348 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:00.249166 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:00.249389 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:00.256459 26176 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:32937
I20260928 23:55:00.256484 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:32937 every 8 connection(s)
I20260928 23:55:00.257522 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:00.260190 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26176
I20260928 23:55:00.260355 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:00.260661 20832 external_mini_cluster.cc:1428] Setting key 8af17b54446ee106c7a32231fc683624
I20260928 23:55:00.261308 26243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:00.268065 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea: Bootstrap starting.
I20260928 23:55:00.270643 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:00.262229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.20.88.62:32937)
0928 23:55:00.262959 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:00.262976 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:00.263013 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:55:00.263709 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:00.263728 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:00.264159 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:00.264590 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:00.265062 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.266056 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.267014 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.267542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.267735 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.268718 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:00.268724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:00.268737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:00.269163 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:00.269225 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:00.269245 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:00.269334 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:00.269536 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:55:00.269853 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260928 23:55:00.270795 26243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:00.271538 26243 log.cc:824] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:00.273838 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:55:00.273981 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea: No bootstrap required, opened a new log
I20260928 23:55:00.277993 26243 raft_consensus.cc:348] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } }
I20260928 23:55:00.278240 26243 raft_consensus.cc:374] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:00.278285 26243 raft_consensus.cc:729] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c642507377314374ab8675823ba62aea, State: Initialized, Role: FOLLOWER
I20260928 23:55:00.278661 26243 consensus_queue.cc:260] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } }
I20260928 23:55:00.278771 26243 raft_consensus.cc:388] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:00.278810 26243 raft_consensus.cc:482] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:00.278910 26243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:00.279915 26243 raft_consensus.cc:504] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } }
I20260928 23:55:00.280339 26243 leader_election.cc:302] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c642507377314374ab8675823ba62aea; no voters:
I20260928 23:55:00.280714 26243 leader_election.cc:288] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:00.280791 26248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:00.280988 26248 raft_consensus.cc:686] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [term 1 LEADER]: Becoming Leader. State: Replica: c642507377314374ab8675823ba62aea, State: Running, Role: LEADER
I20260928 23:55:00.281294 26248 consensus_queue.cc:237] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } }
I20260928 23:55:00.282100 26243 sys_catalog.cc:564] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:00.283067 26250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c642507377314374ab8675823ba62aea. Latest consensus state: current_term: 1 leader_uuid: "c642507377314374ab8675823ba62aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } } }
I20260928 23:55:00.283217 26250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:00.283530 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c642507377314374ab8675823ba62aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c642507377314374ab8675823ba62aea" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32937 } } }
I20260928 23:55:00.283628 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:00.283638 26257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:00.287128 26257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:00.293648 26257 catalog_manager.cc:1347] Generated new cluster ID: 7fe066d0c6e94cceaa5a4ceb6150abeb
I20260928 23:55:00.293744 26257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:00.313573 26257 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:00.314718 26257 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:00.320744 26257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c642507377314374ab8675823ba62aea: Generated new TSK 0
I20260928 23:55:00.321770 26257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:55:00.341150 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:00.335386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.20.88.62:32937)
0928 23:55:00.335535 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:00.335541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:00.335661 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:55:00.335854 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:00.335858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:00.335924 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:00.336055 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:55:00.336317 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.337170 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.338267 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.338642 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.338752 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.339061 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:55:00.340725 (+ 1664us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:55:00.340986 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35870: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:55:00.341233 26246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35870: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 23:55:00.343190 26272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:00.335273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32937 (local address 127.0.0.1:35870)
0928 23:55:00.335553 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:00.335571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:00.335695 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:00.336061 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:00.336063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:00.336070 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:55:00.336172 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:00.336177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.337310 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.337314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:00.338139 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:00.338157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.338775 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.338778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:00.338846 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.338872 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:55:00.340917 (+ 2045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:55:00.342972 (+ 2055us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.88.62:32937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44005/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":1786,"thread_start_us":262,"threads_started":1}
I20260928 23:55:00.343865 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26176
E20260928 23:55:00.356159 26273 webserver.cc:533] Webserver: sslize failed:
E20260928 23:55:00.356317 26273 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T23:55:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T23:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:00Z Disabled control of system clock
I20260928 23:55:00.395143 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32873
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:35055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:32873
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/testchainca.cert with env {}
W20260928 23:55:00.511898 26277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:00.512255 26277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:00.512302 26277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:00.516600 26277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:00.516685 26277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:00.516700 26277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:00.516717 26277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:00.516733 26277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:55:00.516750 26277 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:55:00.516764 26277 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 23:55:00.522733 26277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:32873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:00.523922 26277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:00.524981 26277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:00.534304 26282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:00.534286 26283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:00.534490 26277 server_base.cc:1034] running on GCE node
W20260928 23:55:00.534288 26285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:00.535292 26277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:00.536670 26277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:00.537904 26277 hybrid_clock.cc:648] HybridClock initialized: now 1790639700537832 us; error 82 us; skew 500 ppm
I20260928 23:55:00.540055 26277 webserver.cc:460] Webserver started at http://127.20.88.62:34807/ using document root <none> and password file <none>
I20260928 23:55:00.540779 26277 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:00.540882 26277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:00.541129 26277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:00.543354 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "440cd7043c7c49b0935d11b9c8f7fa52"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "5a12885b4df811c8582f13cfbba2e3d2"
server_key_iv: "73fa6cb1a2adb3a8aa8946954a0284bf"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.543979 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "440cd7043c7c49b0935d11b9c8f7fa52"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "5a12885b4df811c8582f13cfbba2e3d2"
server_key_iv: "73fa6cb1a2adb3a8aa8946954a0284bf"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.548041 26277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:55:00.550978 26291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:00.552624 26277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:00.552820 26277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "440cd7043c7c49b0935d11b9c8f7fa52"
format_stamp: "Formatted at 2026-09-28 23:55:00 on dist-test-slave-wvpw"
server_key: "5a12885b4df811c8582f13cfbba2e3d2"
server_key_iv: "73fa6cb1a2adb3a8aa8946954a0284bf"
server_key_version: "encryptionkey@0"
I20260928 23:55:00.553011 26277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:00.567312 26277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:00.568125 26277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:00.568423 26277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:00.575763 26277 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:32873
I20260928 23:55:00.575820 26343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:32873 every 8 connection(s)
I20260928 23:55:00.576857 26277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:00.579816 26344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:00.581947 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26277
I20260928 23:55:00.582068 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:00.582410 20832 external_mini_cluster.cc:1428] Setting key 7038a27167d23be2720539e59188c9f8
I20260928 23:55:00.586748 26344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52: Bootstrap starting.
I20260928 23:55:00.589211 26344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:00.590008 26344 log.cc:824] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:00.592154 26344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52: No bootstrap required, opened a new log
I20260928 23:55:00.592597 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:00.583922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.20.88.62:32873)
0928 23:55:00.584761 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:00.584779 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:00.584817 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:55:00.585767 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:00.585790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:00.586231 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:00.586617 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:00.587079 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.588083 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.589042 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.589610 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.589782 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.590725 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:00.590731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:00.590745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:00.591155 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:00.591216 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:00.591235 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:00.591352 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:00.591500 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:55:00.591782 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":154,"threads_started":1}
I20260928 23:55:00.595606 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:55:00.596089 26344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } }
I20260928 23:55:00.596388 26344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:00.596467 26344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440cd7043c7c49b0935d11b9c8f7fa52, State: Initialized, Role: FOLLOWER
I20260928 23:55:00.596930 26344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } }
I20260928 23:55:00.597072 26344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:00.597178 26344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:00.597342 26344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:00.598716 26344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } }
I20260928 23:55:00.599097 26344 leader_election.cc:302] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 440cd7043c7c49b0935d11b9c8f7fa52; no voters:
I20260928 23:55:00.599462 26344 leader_election.cc:288] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:00.599664 26349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:00.599932 26349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [term 1 LEADER]: Becoming Leader. State: Replica: 440cd7043c7c49b0935d11b9c8f7fa52, State: Running, Role: LEADER
I20260928 23:55:00.600313 26349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } }
I20260928 23:55:00.600917 26344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:00.602289 26351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 440cd7043c7c49b0935d11b9c8f7fa52. Latest consensus state: current_term: 1 leader_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } } }
I20260928 23:55:00.602289 26350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440cd7043c7c49b0935d11b9c8f7fa52" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32873 } } }
I20260928 23:55:00.602463 26350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:00.602463 26351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:00.602912 26359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:00.606142 26359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:00.612898 26359 catalog_manager.cc:1347] Generated new cluster ID: c1716211c30847338a6258b17144ec9b
I20260928 23:55:00.613000 26359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:00.642640 26359 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:00.643774 26359 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:00.656703 26359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 440cd7043c7c49b0935d11b9c8f7fa52: Generated new TSK 0
I20260928 23:55:00.657639 26359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:55:00.804800 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:00.671691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.20.88.62:32873)
0928 23:55:00.671853 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:00.671859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:00.672039 (+ 180us) server_negotiation.cc:407] Connection header received
0928 23:55:00.672424 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:00.672427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:00.672488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:00.672598 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:55:00.672890 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.673447 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.674731 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.675003 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.675111 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.675498 (+ 387us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:55:00.804344 (+128846us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:55:00.804676 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35246: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 23:55:00.804914 26347 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35246: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 23:55:00.805406 26373 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:00.671582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32873 (local address 127.0.0.1:35246)
0928 23:55:00.671935 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:00.671951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:00.672077 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:00.672635 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:00.672638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:00.672647 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:55:00.672774 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:00.672782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.673601 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.673605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:00.674551 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:00.674563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:00.675168 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:00.675173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:00.675283 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:00.675316 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:55:00.804656 (+129340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:55:00.804870 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.88.62:32873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
I20260928 23:55:00.806267 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26277
2026-09-28T23:55:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (440 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:01 dist-test-slave-wvpw krb5kdc[26385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:01 dist-test-slave-wvpw krb5kdc[26385](info): set up 2 sockets
Sep 28 23:55:01 dist-test-slave-wvpw krb5kdc[26385](info): commencing operation
krb5kdc: starting...
W20260928 23:55:02.137238 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:55:02.276098 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42519
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:02.398079 26398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:02.398396 26398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:02.398450 26398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:02.402731 26398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:02.402803 26398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:02.402823 26398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:02.402841 26398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:02.402858 26398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:02.408465 26398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:02.409560 26398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:02.410590 26398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:02.420079 26406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:02.420091 26403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:02.420212 26398 server_base.cc:1034] running on GCE node
W20260928 23:55:02.420104 26404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:02.421060 26398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:02.422276 26398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:02.423457 26398 hybrid_clock.cc:648] HybridClock initialized: now 1790639702423440 us; error 29 us; skew 500 ppm
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:02.426666 26398 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:55:02.427726 26398 webserver.cc:460] Webserver started at http://127.20.88.62:38109/ using document root <none> and password file <none>
I20260928 23:55:02.428437 26398 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:02.428539 26398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:02.428793 26398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:02.431125 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "6ebcd7c3039849a68fc8d818dcb8b296"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "fdc593985aca6a9bcde9e5d97faf62d3"
server_key_iv: "b8dde8fcf649b5eba018f6ee586a568f"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.431782 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "6ebcd7c3039849a68fc8d818dcb8b296"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "fdc593985aca6a9bcde9e5d97faf62d3"
server_key_iv: "b8dde8fcf649b5eba018f6ee586a568f"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.436110 26398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260928 23:55:02.439071 26413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:02.440644 26398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:02.440834 26398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "6ebcd7c3039849a68fc8d818dcb8b296"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "fdc593985aca6a9bcde9e5d97faf62d3"
server_key_iv: "b8dde8fcf649b5eba018f6ee586a568f"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.440982 26398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:02.459499 26398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:02.463124 26398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:02.463356 26398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:02.471199 26398 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42519
I20260928 23:55:02.471215 26465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42519 every 8 connection(s)
I20260928 23:55:02.472289 26398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:02.474114 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26398
I20260928 23:55:02.474248 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:02.474593 20832 external_mini_cluster.cc:1428] Setting key d7efb9b270e040b1e7c3cff3558548f9
I20260928 23:55:02.475665 26466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:02.481927 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: Bootstrap starting.
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:02.485293 26466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:02.486259 26466 log.cc:824] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:02.488602 26466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: No bootstrap required, opened a new log
I20260928 23:55:02.490521 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:02.476221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.20.88.62:42519)
0928 23:55:02.476718 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:02.476735 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:02.476773 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:55:02.477471 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:02.477489 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:02.477836 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:02.478192 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:02.479077 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.480096 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.481023 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.481546 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.481719 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:02.484786 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:02.484792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:02.484810 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:02.484847 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:02.486946 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:02.487498 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:02.487503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:02.487509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:02.487642 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:02.488005 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:02.488009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:02.488011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:02.488538 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:02.488761 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:02.489098 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 23:55:02.489471 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":158,"threads_started":1}
I20260928 23:55:02.492282 26466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } }
I20260928 23:55:02.492599 26466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:02.492651 26466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebcd7c3039849a68fc8d818dcb8b296, State: Initialized, Role: FOLLOWER
I20260928 23:55:02.493160 26466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } }
I20260928 23:55:02.493323 26466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:02.493368 26466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:02.493448 26466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:02.495023 26466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } }
I20260928 23:55:02.495472 26466 leader_election.cc:302] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ebcd7c3039849a68fc8d818dcb8b296; no voters:
I20260928 23:55:02.495781 26466 leader_election.cc:288] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:02.496006 26471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:02.496498 26471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [term 1 LEADER]: Becoming Leader. State: Replica: 6ebcd7c3039849a68fc8d818dcb8b296, State: Running, Role: LEADER
I20260928 23:55:02.496973 26471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } }
I20260928 23:55:02.497205 26466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:02.499193 26472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } } }
I20260928 23:55:02.499193 26473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ebcd7c3039849a68fc8d818dcb8b296. Latest consensus state: current_term: 1 leader_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebcd7c3039849a68fc8d818dcb8b296" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42519 } } }
I20260928 23:55:02.499377 26472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:02.499377 26473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296 [sys.catalog]: This master's current role is: LEADER
W20260928 23:55:02.502504 26485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:55:02.502631 26485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:55:02.502719 26486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:02.503517 26486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:02.509099 26486 catalog_manager.cc:1347] Generated new cluster ID: 0a5a652894c842ea84c6453fdf41d7b9
I20260928 23:55:02.509279 26486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:02.515439 26486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:02.516541 26486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:02.522091 26486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ebcd7c3039849a68fc8d818dcb8b296: Generated new TSK 0
I20260928 23:55:02.522814 26486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:55:02.556718 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:02.678936 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:02.679250 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:02.679296 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:02.683892 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:02.683979 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:02.684043 26492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:02.689941 26492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:02.691207 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:02.692361 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:02.703015 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:02.703015 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:02.703019 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:02.703301 26492 server_base.cc:1034] running on GCE node
I20260928 23:55:02.703903 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:02.704775 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:02.705950 26492 hybrid_clock.cc:648] HybridClock initialized: now 1790639702705922 us; error 45 us; skew 500 ppm
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:02.709162 26492 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:55:02.710196 26492 webserver.cc:460] Webserver started at http://127.20.88.1:35779/ using document root <none> and password file <none>
I20260928 23:55:02.710831 26492 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:02.710899 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:02.711134 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:02.713516 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "6c1730074036497f88906e3039165513"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "1a884c7b6b18d0b2a1cda4fffae58ef3"
server_key_iv: "0a3e0180ca90d585fee5ba2dcd2075df"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.714148 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "6c1730074036497f88906e3039165513"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "1a884c7b6b18d0b2a1cda4fffae58ef3"
server_key_iv: "0a3e0180ca90d585fee5ba2dcd2075df"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.718397 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:55:02.721362 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:02.722729 26492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:55:02.722889 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "6c1730074036497f88906e3039165513"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "1a884c7b6b18d0b2a1cda4fffae58ef3"
server_key_iv: "0a3e0180ca90d585fee5ba2dcd2075df"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.723024 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:02.747848 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:02.751276 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:02.751536 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:02.752159 26492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:02.753095 26492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:02.753166 26492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:02.753239 26492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:02.753283 26492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:02.762799 26492 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:45475
I20260928 23:55:02.762820 26620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:45475 every 8 connection(s)
I20260928 23:55:02.763796 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:02.764693 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26492
I20260928 23:55:02.764828 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:02.765180 20832 external_mini_cluster.cc:1428] Setting key 30a266514132fa988be78ed5d0cfa4d9
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:02.786020 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:02.766042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:44543 (local address 127.20.88.62:42519)
0928 23:55:02.766193 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:02.766198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:02.768737 (+ 2539us) server_negotiation.cc:407] Connection header received
0928 23:55:02.770112 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:02.770115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:02.770167 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:02.770288 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:02.772174 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.772803 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.774773 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.775034 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.775126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:02.781252 (+ 6126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:02.781257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:02.781260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:02.781302 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:02.783216 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:02.783891 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:02.783898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:02.783902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:02.784016 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:02.784567 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:02.784573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:02.784577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:02.784859 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:02.784988 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:02.785580 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 23:55:02.785745 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:55:02.786799 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:02.766512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42519 (local address 127.20.88.1:44543)
0928 23:55:02.768559 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:02.768612 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:02.769875 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:02.770537 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:02.770551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:02.771170 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:02.771958 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:02.771984 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.772965 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.772971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:02.774608 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:02.774624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:02.775235 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:02.775242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:02.775486 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:02.776222 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:02.776262 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:02.780985 (+ 4723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:02.783390 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:02.783398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:02.783416 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:02.783759 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:02.784153 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:02.784156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:02.784160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:02.784460 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:02.784991 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:02.784997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:02.785289 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 23:55:02.785463 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 23:55:02.785779 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"thread_start_us":122,"threads_started":1}
I20260928 23:55:02.788305 26621 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42519
I20260928 23:55:02.788615 26621 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:02.789184 26621 heartbeater.cc:511] Master 127.20.88.62:42519 requested a full tablet report, sending...
I20260928 23:55:02.790987 26428 ts_manager.cc:194] Registered new tserver with Master: 6c1730074036497f88906e3039165513 (127.20.88.1:45475)
I20260928 23:55:02.792491 26428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:44543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:55:02.800071 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:02.919587 26626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:02.919950 26626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:02.920033 26626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:02.924265 26626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:02.924381 26626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:02.924510 26626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:02.930265 26626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:02.931442 26626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:02.932520 26626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:02.942678 26632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:02.942672 26634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:02.942668 26631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:02.943117 26626 server_base.cc:1034] running on GCE node
I20260928 23:55:02.943621 26626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:02.944489 26626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:02.945672 26626 hybrid_clock.cc:648] HybridClock initialized: now 1790639702945633 us; error 52 us; skew 500 ppm
Sep 28 23:55:02 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:02.948734 26626 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:55:02.949641 26626 webserver.cc:460] Webserver started at http://127.20.88.2:33807/ using document root <none> and password file <none>
I20260928 23:55:02.950214 26626 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:02.950275 26626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:02.950450 26626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:02.952700 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "89d404fbda4a428bb281cb66a8750ab5"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "3a77ff3a329061117a56f24bf0aa7d85"
server_key_iv: "c514642a555f9a870f599bbd5902ae28"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.953259 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "89d404fbda4a428bb281cb66a8750ab5"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "3a77ff3a329061117a56f24bf0aa7d85"
server_key_iv: "c514642a555f9a870f599bbd5902ae28"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.957387 26626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:55:02.960381 26641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:02.961833 26626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:02.961958 26626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "89d404fbda4a428bb281cb66a8750ab5"
format_stamp: "Formatted at 2026-09-28 23:55:02 on dist-test-slave-wvpw"
server_key: "3a77ff3a329061117a56f24bf0aa7d85"
server_key_iv: "c514642a555f9a870f599bbd5902ae28"
server_key_version: "encryptionkey@0"
I20260928 23:55:02.962076 26626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:03.004426 26626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:03.008049 26626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:03.008279 26626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:03.009199 26626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:03.010119 26626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:03.010171 26626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:03.010214 26626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:03.010231 26626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:03.019444 26626 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:46259
I20260928 23:55:03.019462 26754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:46259 every 8 connection(s)
I20260928 23:55:03.020466 26626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:03.027789 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26626
I20260928 23:55:03.027927 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:03.028213 20832 external_mini_cluster.cc:1428] Setting key 105dd51018ba4b3b507cd861da8057af
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:03.036134 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.022442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:36667 (local address 127.20.88.62:42519)
0928 23:55:03.022634 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.022638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.023425 (+ 787us) server_negotiation.cc:407] Connection header received
0928 23:55:03.024993 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.024997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.025078 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.025182 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.026408 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.027006 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.028521 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.028770 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.028890 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.032035 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:03.032038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:03.032041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:03.032083 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:03.033836 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.034473 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.034479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.034481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.034572 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.034935 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.034940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.034941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.035163 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:03.035270 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:03.035821 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 23:55:03.035956 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 23:55:03.037180 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.022805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42519 (local address 127.20.88.2:36667)
0928 23:55:03.023267 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:03.023312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:03.024769 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:03.025374 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:03.025381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:03.025761 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:03.026260 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:03.026274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.027144 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.027148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:03.028392 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:03.028402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.028935 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.028939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:03.029150 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.029880 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:03.029897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:03.031836 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:03.033970 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:03.033976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:03.033996 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:03.034359 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:03.034670 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:03.034675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:03.034677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:03.034837 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:03.035284 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:03.035292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:03.035576 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 23:55:03.035780 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 23:55:03.036138 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260928 23:55:03.038484 26755 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42519
I20260928 23:55:03.038760 26755 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:55:03.039333 26755 heartbeater.cc:511] Master 127.20.88.62:42519 requested a full tablet report, sending...
I20260928 23:55:03.040611 26428 ts_manager.cc:194] Registered new tserver with Master: 89d404fbda4a428bb281cb66a8750ab5 (127.20.88.2:46259)
I20260928 23:55:03.041349 26428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:36667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:55:03.060109 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:32815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:03.178406 26760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:03.178701 26760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:03.178746 26760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:03.182875 26760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:03.182950 26760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:03.183012 26760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:03.191780 26760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:32815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:03.193375 26760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:03.194499 26760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:03.205108 26768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:03.205120 26766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:03.205129 26765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:03.205353 26760 server_base.cc:1034] running on GCE node
I20260928 23:55:03.205950 26760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:03.206666 26760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:03.207862 26760 hybrid_clock.cc:648] HybridClock initialized: now 1790639703207846 us; error 28 us; skew 500 ppm
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.211206 26760 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:55:03.212354 26760 webserver.cc:460] Webserver started at http://127.20.88.3:37297/ using document root <none> and password file <none>
I20260928 23:55:03.213018 26760 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:03.213088 26760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:03.213264 26760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:03.215447 26760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "8a8d7b164e774af5bfce35a414365349"
format_stamp: "Formatted at 2026-09-28 23:55:03 on dist-test-slave-wvpw"
server_key: "f12406a994be8b6b9f624fb9f8560329"
server_key_iv: "c24ef9759d4061a4223fde09f77dcc17"
server_key_version: "encryptionkey@0"
I20260928 23:55:03.215993 26760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "8a8d7b164e774af5bfce35a414365349"
format_stamp: "Formatted at 2026-09-28 23:55:03 on dist-test-slave-wvpw"
server_key: "f12406a994be8b6b9f624fb9f8560329"
server_key_iv: "c24ef9759d4061a4223fde09f77dcc17"
server_key_version: "encryptionkey@0"
I20260928 23:55:03.220126 26760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:03.223003 26775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:03.224581 26760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:03.224733 26760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "8a8d7b164e774af5bfce35a414365349"
format_stamp: "Formatted at 2026-09-28 23:55:03 on dist-test-slave-wvpw"
server_key: "f12406a994be8b6b9f624fb9f8560329"
server_key_iv: "c24ef9759d4061a4223fde09f77dcc17"
server_key_version: "encryptionkey@0"
I20260928 23:55:03.224853 26760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:03.241428 26760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:03.245091 26760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:03.245313 26760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:03.245952 26760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:03.246774 26760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:03.246821 26760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:03.246896 26760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:03.246930 26760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:03.255760 26760 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:38059
I20260928 23:55:03.255784 26888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:38059 every 8 connection(s)
I20260928 23:55:03.256717 26760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:03.257292 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26760
I20260928 23:55:03.257426 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:03.257714 20832 external_mini_cluster.cc:1428] Setting key db0e2c83be94a141b5486593d27c2903
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:03.271643 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.258728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:58639 (local address 127.20.88.62:42519)
0928 23:55:03.258857 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.258862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.259876 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 23:55:03.261434 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.261437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.261485 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.261554 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.262762 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.263296 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.264536 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.264771 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.264862 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.267709 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:03.267713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:03.267715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:03.267743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:03.269278 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.269806 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.269809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.269810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.269899 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.270245 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.270250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.270252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.270561 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:03.270696 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:03.271347 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 23:55:03.271478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:55:03.272744 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.259172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42519 (local address 127.20.88.3:58639)
0928 23:55:03.259724 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:03.259784 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:03.261208 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:03.261724 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:03.261730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:03.262087 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:03.262626 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:03.262639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.263400 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.263403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:03.264406 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:03.264415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.264974 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.264978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:03.265139 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.265748 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:03.265765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:03.267503 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:03.269398 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:03.269405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:03.269419 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:03.269708 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:03.269983 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:03.269985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:03.269986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:03.270127 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:03.270692 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:03.270697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:03.271066 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 23:55:03.271309 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 23:55:03.271690 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260928 23:55:03.273849 26889 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42519
I20260928 23:55:03.274134 26889 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:03.274659 26889 heartbeater.cc:511] Master 127.20.88.62:42519 requested a full tablet report, sending...
I20260928 23:55:03.275701 26428 ts_manager.cc:194] Registered new tserver with Master: 8a8d7b164e774af5bfce35a414365349 (127.20.88.3:38059)
I20260928 23:55:03.276437 26428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:58639
I20260928 23:55:03.284039 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/test-admin.keytab.
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.302399 20832 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 23:55:03.305516 26896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:03.303982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.20.88.1:45475)
0928 23:55:03.304367 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.304373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.304398 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:55:03.304451 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.304457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.304656 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.304785 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.305323 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38500: BlockingRecv error: recv got EOF from 127.0.0.1:38500 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
W20260928 23:55:03.305631 20832 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.306828 20832 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.307842 20832 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:55:03.326630 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.308962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.20.88.1:45475)
0928 23:55:03.309103 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.309106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.309141 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:55:03.309265 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.309268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.309321 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.309396 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.310016 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.310910 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.311813 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.312324 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.312478 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.322215 (+ 9737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:03.322233 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:03.322251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:03.322295 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:03.324380 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.324959 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.324966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.324967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.325068 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.325432 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.325436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.325437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.325767 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:03.325949 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:03.326251 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 23:55:03.326492 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:55:03.331321 26896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:03.330130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.20.88.1:45475)
0928 23:55:03.330254 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.330258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.330360 (+ 102us) server_negotiation.cc:407] Connection header received
0928 23:55:03.330523 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.330526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.330578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.330678 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.331185 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38520: BlockingRecv error: recv got EOF from 127.0.0.1:38520 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:55:03.331532 20832 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.332769 20832 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.333871 20832 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:55:03.344673 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.334819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.20.88.1:45475)
0928 23:55:03.335216 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.335221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.335236 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:03.335330 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.335333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.335408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.335536 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.336262 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.337110 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.338481 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.338711 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.338803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.341586 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:03.341589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:03.341591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:03.341621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:03.342966 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.343411 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.343413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.343414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.343494 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.343812 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.343815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.343816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.344012 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:03.344129 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:03.344408 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 23:55:03.344547 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 23:55:03.347676 26896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:03.346688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.20.88.1:45475)
0928 23:55:03.346868 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.346872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.346912 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:55:03.347054 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.347056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.347095 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.347156 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.347556 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260928 23:55:03.347725 20832 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639669102918-20832-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.348865 20832 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:03.349756 20832 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:55:03.358397 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:03.350723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.20.88.1:45475)
0928 23:55:03.350872 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:03.350875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:03.350888 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:55:03.351011 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:03.351013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:03.351053 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:03.351120 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:03.351584 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.352066 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.352873 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:03.353077 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:03.353153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:03.355412 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:03.355415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:03.355417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:03.355440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:03.356948 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.357325 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.357328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.357329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.357397 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:03.357616 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:03.357618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:03.357619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:03.357776 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:03.357886 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:03.358088 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:55:03.358178 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:55:03.359724 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26492
I20260928 23:55:03.366089 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26626
I20260928 23:55:03.371714 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26760
I20260928 23:55:03.377449 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26398
2026-09-28T23:55:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2311 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26909](info): set up 2 sockets
Sep 28 23:55:03 dist-test-slave-wvpw krb5kdc[26909](info): commencing operation
krb5kdc: starting...
W20260928 23:55:04.429284 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:04 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab.
Entry for principal oryx/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab.
I20260928 23:55:04.559484 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41035
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab
--principal=oryx/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:04.696547 26922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:04.696914 26922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:04.696990 26922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:04.701299 26922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:04.701429 26922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:04.701473 26922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:04.701536 26922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:04.701575 26922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:04.707502 26922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab
--principal=oryx/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:04.708855 26922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:04.709986 26922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:04.719391 26930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:04.719377 26927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:04.719379 26928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:04.719468 26922 server_base.cc:1034] running on GCE node
I20260928 23:55:04.720431 26922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:04.721640 26922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:04.722817 26922 hybrid_clock.cc:648] HybridClock initialized: now 1790639704722800 us; error 31 us; skew 500 ppm
Sep 28 23:55:04 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:04.725891 26922 init.cc:377] Logged in from keytab as oryx/127.20.88.62@KRBTEST.COM (short username oryx)
I20260928 23:55:04.726974 26922 webserver.cc:460] Webserver started at http://127.20.88.62:37367/ using document root <none> and password file <none>
I20260928 23:55:04.727594 26922 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:04.727691 26922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:04.727942 26922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:04.730216 26922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "0591079a0cfe4a50a667a39a1ea6c23d"
format_stamp: "Formatted at 2026-09-28 23:55:04 on dist-test-slave-wvpw"
server_key: "ca3744e44f3ea9f235f174bc3e858e76"
server_key_iv: "5febbefaeacb1ee41d5315571a880e19"
server_key_version: "encryptionkey@0"
I20260928 23:55:04.730840 26922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "0591079a0cfe4a50a667a39a1ea6c23d"
format_stamp: "Formatted at 2026-09-28 23:55:04 on dist-test-slave-wvpw"
server_key: "ca3744e44f3ea9f235f174bc3e858e76"
server_key_iv: "5febbefaeacb1ee41d5315571a880e19"
server_key_version: "encryptionkey@0"
I20260928 23:55:04.734934 26922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 23:55:04.737656 26937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:04.739426 26922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:55:04.739616 26922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "0591079a0cfe4a50a667a39a1ea6c23d"
format_stamp: "Formatted at 2026-09-28 23:55:04 on dist-test-slave-wvpw"
server_key: "ca3744e44f3ea9f235f174bc3e858e76"
server_key_iv: "5febbefaeacb1ee41d5315571a880e19"
server_key_version: "encryptionkey@0"
I20260928 23:55:04.739760 26922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:04.763731 26922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:04.767236 26922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:04.767493 26922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:04.775408 26922 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:41035
I20260928 23:55:04.775746 26989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:41035 every 8 connection(s)
I20260928 23:55:04.776598 26922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:04.776971 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 26922
I20260928 23:55:04.777140 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:04.777490 20832 external_mini_cluster.cc:1428] Setting key e01d6ece651483d81fdb5e9614afa45c
I20260928 23:55:04.780007 26990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:04.786115 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Bootstrap starting.
Sep 28 23:55:04 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:04.789073 26990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:04.790068 26990 log.cc:824] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:04.792608 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: No bootstrap required, opened a new log
I20260928 23:55:04.792835 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:04.778917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.20.88.62:41035)
0928 23:55:04.779360 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:04.779380 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:04.779649 (+ 269us) server_negotiation.cc:407] Connection header received
0928 23:55:04.780587 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:04.780606 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:04.780955 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:04.781313 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:04.782131 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:04.783151 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:04.784033 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:04.784594 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:04.784777 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:04.787423 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:04.787430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:04.787452 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:04.787492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:04.789600 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:04.790068 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:04.790075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:04.790083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:04.790231 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:04.790510 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:04.790515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:04.790517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:04.791024 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:04.791207 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:04.791492 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 23:55:04.791804 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":155,"threads_started":1}
I20260928 23:55:04.796649 26990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } }
I20260928 23:55:04.796887 26990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:04.796932 26990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0591079a0cfe4a50a667a39a1ea6c23d, State: Initialized, Role: FOLLOWER
I20260928 23:55:04.797326 26990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } }
I20260928 23:55:04.797451 26990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:04.797497 26990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:04.797567 26990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:04.798496 26990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } }
I20260928 23:55:04.798902 26990 leader_election.cc:302] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0591079a0cfe4a50a667a39a1ea6c23d; no voters:
I20260928 23:55:04.799288 26990 leader_election.cc:288] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:04.799396 26995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:04.799573 26995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 LEADER]: Becoming Leader. State: Replica: 0591079a0cfe4a50a667a39a1ea6c23d, State: Running, Role: LEADER
I20260928 23:55:04.799883 26995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } }
I20260928 23:55:04.800731 26990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:04.801751 26996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } }
I20260928 23:55:04.801892 26996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:04.802239 26997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0591079a0cfe4a50a667a39a1ea6c23d. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } }
I20260928 23:55:04.802302 27004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:04.802397 26997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:04.805835 27004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:04.812167 27004 catalog_manager.cc:1347] Generated new cluster ID: 8d77d4c16fbe4e2e9d25b5f59a67af22
I20260928 23:55:04.812299 27004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:04.825129 27004 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:04.826542 27004 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:04.835886 27004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Generated new TSK 0
I20260928 23:55:04.836900 27004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
I20260928 23:55:04.880712 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:05.025242 27016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:05.025602 27016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:05.025686 27016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:05.030157 27016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:05.030270 27016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:05.030370 27016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:05.036120 27016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:05.037432 27016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:05.038592 27016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:05.049273 27022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.049273 27024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.049299 27021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:05.049767 27016 server_base.cc:1034] running on GCE node
I20260928 23:55:05.050246 27016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:05.050968 27016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:05.052165 27016 hybrid_clock.cc:648] HybridClock initialized: now 1790639705052146 us; error 37 us; skew 500 ppm
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:05.055126 27016 init.cc:377] Logged in from keytab as oryx/127.20.88.1@KRBTEST.COM (short username oryx)
I20260928 23:55:05.055997 27016 webserver.cc:460] Webserver started at http://127.20.88.1:44949/ using document root <none> and password file <none>
I20260928 23:55:05.056615 27016 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:05.056676 27016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:05.056842 27016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:05.059005 27016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "56800e01ac454625836872974887c3e3"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "b8d51202e8d037dd40d01a95b300af71"
server_key_iv: "5a4a70f7fa738470a4531bd44d039f9a"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.059540 27016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "56800e01ac454625836872974887c3e3"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "b8d51202e8d037dd40d01a95b300af71"
server_key_iv: "5a4a70f7fa738470a4531bd44d039f9a"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.063431 27016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:05.066174 27031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.067636 27016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:05.067786 27016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "56800e01ac454625836872974887c3e3"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "b8d51202e8d037dd40d01a95b300af71"
server_key_iv: "5a4a70f7fa738470a4531bd44d039f9a"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.067915 27016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:05.085952 27016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:05.089603 27016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:05.089808 27016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:05.090442 27016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:05.091285 27016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:05.091356 27016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.091439 27016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:05.091470 27016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.100773 27016 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:40145
I20260928 23:55:05.100831 27144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:40145 every 8 connection(s)
I20260928 23:55:05.101894 27016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:05.109301 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27016
I20260928 23:55:05.109469 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:05.109750 20832 external_mini_cluster.cc:1428] Setting key 92ff3828c2fa1df76afa30bf992a855b
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:05.118507 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.104163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:37281 (local address 127.20.88.62:41035)
0928 23:55:05.104400 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.104404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.105416 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 23:55:05.106868 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.106872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.106946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.107059 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.108395 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.108960 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.110307 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.110593 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.110704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.113861 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.113867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.113870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.113907 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.115843 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.116590 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.116596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.116601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.116722 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.117135 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.117143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.117146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.117461 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.117582 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.118127 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.118294 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 23:55:05.119387 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.104640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:37281)
0928 23:55:05.105217 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.105266 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.106626 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.107256 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.107263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.107665 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.108192 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.108206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.109105 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.109109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.110168 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.110179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.110755 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.110758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.110964 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.111701 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.111719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.113646 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.116033 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.116043 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.116065 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.116458 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.116832 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.116834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.116836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.116990 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.117579 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.117584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.117850 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.118027 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.118362 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for oryx/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:55:05.120932 27145 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:05.121202 27145 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:05.121799 27145 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.2@KRBTEST.COM" created.
I20260928 23:55:05.123703 26954 ts_manager.cc:194] Registered new tserver with Master: 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:05.124933 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:37281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
I20260928 23:55:05.140045 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:05.256362 27150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:05.256703 27150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:05.256803 27150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:05.261334 27150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:05.261463 27150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:05.261564 27150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:05.267617 27150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:05.268993 27150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:05.270355 27150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:05.281270 27156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.281263 27155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.281268 27158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:05.281677 27150 server_base.cc:1034] running on GCE node
I20260928 23:55:05.282140 27150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:05.282872 27150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:05.284106 27150 hybrid_clock.cc:648] HybridClock initialized: now 1790639705284064 us; error 36 us; skew 500 ppm
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:05.287155 27150 init.cc:377] Logged in from keytab as oryx/127.20.88.2@KRBTEST.COM (short username oryx)
I20260928 23:55:05.288334 27150 webserver.cc:460] Webserver started at http://127.20.88.2:41087/ using document root <none> and password file <none>
I20260928 23:55:05.288975 27150 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:05.289078 27150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:05.289342 27150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:05.291631 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "e6528b323b0540a8b6ce27d2068abce9"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "032bc4a52459a6dd958f0aa7d3895914"
server_key_iv: "6f0e8c6c46b1442648fac32484883344"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.292531 27150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "e6528b323b0540a8b6ce27d2068abce9"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "032bc4a52459a6dd958f0aa7d3895914"
server_key_iv: "6f0e8c6c46b1442648fac32484883344"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.296958 27150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:05.299819 27165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.301594 27150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:05.301780 27150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "e6528b323b0540a8b6ce27d2068abce9"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "032bc4a52459a6dd958f0aa7d3895914"
server_key_iv: "6f0e8c6c46b1442648fac32484883344"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.301961 27150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:05.316365 27150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:05.319871 27150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:05.320130 27150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:05.320844 27150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:05.321761 27150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:05.321839 27150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.321913 27150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:05.321959 27150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.331059 27150 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:42383
I20260928 23:55:05.331173 27278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:42383 every 8 connection(s)
I20260928 23:55:05.332044 27150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:05.337235 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27150
I20260928 23:55:05.337364 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:05.337692 20832 external_mini_cluster.cc:1428] Setting key 2901ee8f0e738cf7bfa5208df9a3733e
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:55:05.349221 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.334038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:34607 (local address 127.20.88.62:41035)
0928 23:55:05.334210 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.334214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.335193 (+ 979us) server_negotiation.cc:407] Connection header received
0928 23:55:05.336719 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.336722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.336773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.336850 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.338375 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.339025 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.340468 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.340884 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.341029 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.344345 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.344350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.344354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.344385 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.346333 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.347196 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.347203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.347205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.347323 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.347760 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.347766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.347768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.348067 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.348184 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.348839 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.349005 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:55:05.350131 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.334437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:34607)
0928 23:55:05.335005 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.335069 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.336500 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.337037 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.337045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.337541 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.338222 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.338247 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.339170 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.339174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.340340 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.340349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.341105 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.341112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.341381 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.342177 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.342205 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.344090 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.346496 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.346503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.346525 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.346960 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.347448 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.347450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.347452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.347629 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.348190 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.348196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.348555 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.348723 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.349079 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260928 23:55:05.351516 27279 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:05.351800 27279 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.3@KRBTEST.COM" created.
I20260928 23:55:05.352454 27279 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
I20260928 23:55:05.353765 26954 ts_manager.cc:194] Registered new tserver with Master: e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383)
I20260928 23:55:05.354418 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:34607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
I20260928 23:55:05.371018 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:05.489210 27284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:05.489545 27284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:05.489593 27284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:05.493829 27284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:05.493906 27284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:05.493968 27284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:05.499718 27284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:05.501025 27284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:05.502233 27284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:05.512645 27289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.512594 27290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.512609 27292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:05.512876 27284 server_base.cc:1034] running on GCE node
I20260928 23:55:05.513276 27284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:05.513962 27284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:05.515168 27284 hybrid_clock.cc:648] HybridClock initialized: now 1790639705515155 us; error 27 us; skew 500 ppm
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:05.518064 27284 init.cc:377] Logged in from keytab as oryx/127.20.88.3@KRBTEST.COM (short username oryx)
I20260928 23:55:05.519001 27284 webserver.cc:460] Webserver started at http://127.20.88.3:38181/ using document root <none> and password file <none>
I20260928 23:55:05.519630 27284 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:05.519726 27284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:05.519963 27284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:05.522203 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "bc8bf8e430a3be35ebf8bf7e5a511aa3"
server_key_iv: "6625d6c0467029d98fe26dc2145ebf11"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.522805 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "bc8bf8e430a3be35ebf8bf7e5a511aa3"
server_key_iv: "6625d6c0467029d98fe26dc2145ebf11"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.527293 27284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:05.530205 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.531605 27284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:05.531769 27284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "bc8bf8e430a3be35ebf8bf7e5a511aa3"
server_key_iv: "6625d6c0467029d98fe26dc2145ebf11"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.531909 27284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:05.622861 27284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:05.626220 27284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:05.626456 27284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:05.627085 27284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:05.627978 27284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:05.628049 27284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.628145 27284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:05.628196 27284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.637422 27284 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:43281
I20260928 23:55:05.637440 27412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:43281 every 8 connection(s)
I20260928 23:55:05.638401 27284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:05.639720 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27284
I20260928 23:55:05.639860 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:05.640165 20832 external_mini_cluster.cc:1428] Setting key 96a1d2ce1a89941fc1d29554707b3089
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:05.654175 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.640444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:35267 (local address 127.20.88.62:41035)
0928 23:55:05.640609 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.640614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.641801 (+ 1187us) server_negotiation.cc:407] Connection header received
0928 23:55:05.643539 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.643545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.643628 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.643707 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.645111 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.645652 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.646921 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.647152 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.647235 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.650059 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.650063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.650065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.650096 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.651750 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.652341 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.652344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.652345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.652439 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.652757 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.652759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.652760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.652974 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.653052 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.653878 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.654050 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:55:05.655189 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.640986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:35267)
0928 23:55:05.641627 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.641693 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.643284 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.643890 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.643898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.644397 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.644965 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.644980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.645800 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.645803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.646796 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.646806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.647297 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.647300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.647482 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.648148 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.648166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.649862 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.651886 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.651892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.651908 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.652200 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.652523 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.652525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.652526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.652670 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.653187 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.653194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.653550 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.653760 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.654170 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
I20260928 23:55:05.656486 27413 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:05.656747 27413 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:05.657272 27413 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
I20260928 23:55:05.658295 26954 ts_manager.cc:194] Registered new tserver with Master: 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
I20260928 23:55:05.658896 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:35267
I20260928 23:55:05.667184 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.88.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.61.keytab.
Entry for principal oryx/127.20.88.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.61.keytab.
I20260928 23:55:05.696206 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:40149
--webserver_interface=127.20.88.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.61.keytab
--principal=oryx/127.20.88.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:41035,127.20.88.61:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:05.811322 27418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:05.811678 27418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:05.811755 27418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:05.815929 27418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:05.816061 27418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:05.816097 27418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:05.816138 27418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:05.816171 27418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:05.821841 27418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:41035,127.20.88.61:40149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.61.keytab
--principal=oryx/127.20.88.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.88.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
W20260928 23:55:05.822176 27418 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.88.62:41035,127.20.88.61:40149'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 23:55:05.823168 27418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:05.824312 27418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:05.833150 27426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.833163 27424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:05.833177 27423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:05.833384 27418 server_base.cc:1034] running on GCE node
I20260928 23:55:05.834028 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:05.835206 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:05.836395 27418 hybrid_clock.cc:648] HybridClock initialized: now 1790639705836385 us; error 33 us; skew 500 ppm
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:05.839272 27418 init.cc:377] Logged in from keytab as oryx/127.20.88.61@KRBTEST.COM (short username oryx)
I20260928 23:55:05.840337 27418 webserver.cc:460] Webserver started at http://127.20.88.61:40991/ using document root <none> and password file <none>
I20260928 23:55:05.840961 27418 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:05.841048 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:05.841281 27418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:05.843384 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/instance:
uuid: "707eaa96c0b24bd2be5f7310ca10c680"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "a7d3dbb043f5098faeb0a044f3f452f6"
server_key_iv: "349bf4b57f11842e0cc9334ff16aab04"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.843954 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal/instance:
uuid: "707eaa96c0b24bd2be5f7310ca10c680"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "a7d3dbb043f5098faeb0a044f3f452f6"
server_key_iv: "349bf4b57f11842e0cc9334ff16aab04"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.847924 27418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:55:05.850764 27433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.852144 27418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:05.852329 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "707eaa96c0b24bd2be5f7310ca10c680"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "a7d3dbb043f5098faeb0a044f3f452f6"
server_key_iv: "349bf4b57f11842e0cc9334ff16aab04"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.852469 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:05.869108 27418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:05.872718 27418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:05.872946 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:05.880627 27418 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.61:40149
I20260928 23:55:05.880959 27485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.61:40149 every 8 connection(s)
I20260928 23:55:05.881749 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
I20260928 23:55:05.883343 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27418
I20260928 23:55:05.885314 27486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:05.889364 27486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.61@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:05.900900 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.885158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.20.88.61:40149)
0928 23:55:05.885656 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.885673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.885713 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:55:05.886367 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.886384 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.886788 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.887167 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.888012 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.889097 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.890110 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.890616 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.890785 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.894893 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.894900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.894909 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.894986 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.897031 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.897641 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.897648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.897658 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.897797 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.898152 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.898157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.898160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.898720 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.898947 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.899302 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.899662 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":153,"threads_started":1}
I20260928 23:55:05.901859 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.891287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.20.88.62:41035)
0928 23:55:05.891455 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.891459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.891728 (+ 269us) server_negotiation.cc:407] Connection header received
0928 23:55:05.891841 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.891844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.891896 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.891985 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.892929 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.893446 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.894555 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.894894 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.894980 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.897780 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.897783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.897785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.897811 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.899653 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.900209 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.900212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.900213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.900385 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.900751 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.900754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.900755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.900980 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.901095 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.901550 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.901697 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:55:05.901872 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.891164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56498)
0928 23:55:05.891605 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.891645 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.891754 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.892091 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.892100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.892596 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.892787 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.892795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.893596 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.893602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.894353 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.894366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.895031 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.895035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.895166 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.895800 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.895817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.897626 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.899789 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.899796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.899803 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.900086 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.900526 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.900529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.900531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.900624 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.901150 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.901157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.901278 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.901490 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.901691 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":166,"threads_started":1}
I20260928 23:55:05.903231 27486 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639705, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.61@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
I20260928 23:55:05.911942 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.903735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48304)
0928 23:55:05.903853 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.903870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.903948 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.904189 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.904192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.904452 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.904569 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.904574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.905723 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.905726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.906252 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.906260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.906770 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.906773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.906847 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.907367 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.907379 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.908749 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.910391 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.910397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.910400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.910600 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.910904 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.910907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.910909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.910992 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.911368 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.911371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.911458 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.911570 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.911728 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:55:05.911942 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.903817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.20.88.61:40149)
0928 23:55:05.903925 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.903928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.903938 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:55:05.904011 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.904013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.904045 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.904110 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.904689 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.905590 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.906373 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.906606 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.906744 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.908901 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.908904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.908906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.908926 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.910274 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.910722 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.910729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.910731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.910808 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.911084 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.911087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.911089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.911268 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.911369 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.911615 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.911809 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":43}
I20260928 23:55:05.915117 27486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Bootstrap starting.
I20260928 23:55:05.917749 27486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:05.918557 27486 log.cc:824] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:05.920750 27486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: No bootstrap required, opened a new log
I20260928 23:55:05.923808 27486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } }
I20260928 23:55:05.924098 27486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:05.924182 27486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707eaa96c0b24bd2be5f7310ca10c680, State: Initialized, Role: FOLLOWER
I20260928 23:55:05.924718 27486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } }
I20260928 23:55:05.925243 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } } }
I20260928 23:55:05.925379 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:05.926034 27486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:55:05.931259 27503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:55:05.931378 27503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:55:05.933830 27418 master_runner.cc:428] Detected that this master 707eaa96c0b24bd2be5f7310ca10c680 is joining an existing cluster
I20260928 23:55:05.933903 27418 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.88.61:40149
I20260928 23:55:05.945752 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.936774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.20.88.61:40149)
0928 23:55:05.936885 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.936891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.937248 (+ 357us) server_negotiation.cc:407] Connection header received
0928 23:55:05.937334 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.937337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.937383 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.937467 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.937981 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.939191 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940008 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.940215 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.942159 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.942163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.942165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.942189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.944024 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944486 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.944489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.944490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.944608 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944850 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.944856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.944858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.945036 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.945141 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.945369 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.945598 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":33}
I20260928 23:55:05.945768 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.936370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.20.88.62:41035)
0928 23:55:05.936520 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.936523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.936759 (+ 236us) server_negotiation.cc:407] Connection header received
0928 23:55:05.936939 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.936942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.937059 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.937137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.937999 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.938871 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940157 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.940552 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940635 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.942043 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.942047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.942052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.942089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.943807 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944386 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.944390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.944393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.944492 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944802 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.944806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.944808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.945057 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.945133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.945470 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.945598 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:55:05.945765 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.936873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48306)
0928 23:55:05.937149 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.937162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.937254 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.937502 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.937506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.937701 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.937869 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.937876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.939317 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.939320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.939884 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.939891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.940374 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.940458 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.941623 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.941635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.942028 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.944132 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.944138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.944403 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.944684 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.944689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.944767 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.945131 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.945134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.945219 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.945328 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.945542 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":48,"thread_start_us":136,"threads_started":1}
I20260928 23:55:05.946548 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.936286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56514)
0928 23:55:05.936663 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.936676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.936808 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.937340 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.937343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.937641 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.937844 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.937855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.939024 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.939027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.940010 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.940021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.940793 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.940796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.940889 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.941355 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.941371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.941873 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.943966 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.943969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.943971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.944280 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.944609 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.944612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.944614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.944702 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.945205 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.945208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.945294 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.946214 (+ 920us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.946367 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":20,"thread_start_us":101,"threads_started":1}
Sep 28 23:55:05 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.62@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.20.88.60@KRBTEST.COM; defaulting to no policy
I20260928 23:55:05.964497 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.951971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48308)
0928 23:55:05.952814 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:05.952854 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:05.952975 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:05.953304 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:05.953309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:05.953837 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:05.953959 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.953965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.955249 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.955254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.956499 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:05.956512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.957257 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.957261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:05.957387 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.958073 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:05.958090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:05.960120 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:05.962390 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.962403 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.962411 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.962788 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.963133 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:05.963138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:05.963143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:05.963246 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:05.963771 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:05.963782 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:05.963935 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 23:55:05.964076 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 23:55:05.964343 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":93,"threads_started":1}
I20260928 23:55:05.964519 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:05.952037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.20.88.61:40149)
0928 23:55:05.952172 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:05.952177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:05.952930 (+ 753us) server_negotiation.cc:407] Connection header received
0928 23:55:05.953104 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:05.953108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:05.953164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:05.953252 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:05.954401 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.955122 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.956735 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:05.957125 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:05.957249 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:05.960329 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:05.960333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:05.960337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:05.960367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:05.962243 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.962913 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.962917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.962920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.963023 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:05.963344 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:05.963347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:05.963349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:05.963643 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:05.963743 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:05.964158 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 23:55:05.964357 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.60@KRBTEST.COM" created.
I20260928 23:55:05.966349 26954 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.88.61:40149: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680"
member_type: NON_VOTER
last_known_addr {
host: "127.20.88.61"
port: 40149
}
attrs {
promote: true
}
}
dest_uuid: "0591079a0cfe4a50a667a39a1ea6c23d"
cas_config_opid_index: -1
I20260928 23:55:05.967100 26954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } }
I20260928 23:55:05.969295 27515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 707eaa96c0b24bd2be5f7310ca10c680 (127.20.88.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } }
I20260928 23:55:05.970343 27460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:05.971837 27460 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 23:55:05.971860 27515 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.88.61:40149
I20260928 23:55:05.972311 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } }
I20260928 23:55:05.972415 27418 master.cc:537] Master@127.20.88.61:40149 shutting down...
I20260928 23:55:05.972445 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:05.972757 27516 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:55:05.975250 26939 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 23:55:05.977074 26939 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d -> Peer 707eaa96c0b24bd2be5f7310ca10c680 (127.20.88.61:40149): Couldn't send request to peer 707eaa96c0b24bd2be5f7310ca10c680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:55:05.977463 27418 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:05.977766 27418 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:05.977896 27418 tablet_replica.cc:330] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.60.keytab.
Entry for principal oryx/127.20.88.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.60.keytab.
I20260928 23:55:05.988204 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:36131
--webserver_interface=127.20.88.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.60.keytab
--principal=oryx/127.20.88.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:55:05.993717 27418 master.cc:552] Master@127.20.88.61:40149 shutdown complete.
I20260928 23:55:05.993803 27418 master_runner.cc:305] Clearing existing system tablet
I20260928 23:55:05.994048 27418 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:05.994123 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:05.995275 27418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:55:05.996179 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:05.996438 27418 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:05.996590 27418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:05.996629 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "707eaa96c0b24bd2be5f7310ca10c680"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "a7d3dbb043f5098faeb0a044f3f452f6"
server_key_iv: "349bf4b57f11842e0cc9334ff16aab04"
server_key_version: "encryptionkey@0"
I20260928 23:55:05.998436 27418 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:06.000746 27418 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:55:06.000869 27418 log.cc:1196] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 23:55:06.001107 27418 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Deleting consensus metadata
I20260928 23:55:06.001389 27418 master_runner.cc:315] Copying system tablet from 127.20.88.62:41035
I20260928 23:55:06.002760 27418 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: tablet copy: Beginning tablet copy session from remote peer at address 127.20.88.62:41035
I20260928 23:55:06.014609 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.003394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56522)
0928 23:55:06.007255 (+ 3861us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.007272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.007491 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.007788 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.007791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.008022 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.008131 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.008137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.008995 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.008999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.009611 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.009619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.010099 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.010103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.010215 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.010771 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.010783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.011126 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.013016 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.013021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.013023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.013233 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.013520 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.013524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.013526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.013613 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.014029 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.014032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.014138 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.014250 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.014417 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3750,"thread_start_us":98,"threads_started":1}
I20260928 23:55:06.014597 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.003457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.20.88.62:41035)
0928 23:55:06.003619 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.003625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.007387 (+ 3762us) server_negotiation.cc:407] Connection header received
0928 23:55:06.007616 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.007620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.007676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.007768 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.008302 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.008831 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.009747 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.009978 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.010092 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.011278 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.011282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.011285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.011319 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.012909 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.013338 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.013342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.013344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.013430 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.013713 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.013716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.013718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.013932 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.014042 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.014315 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.014430 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:55:06.015482 26974 tablet_copy_service.cc:140] P 0591079a0cfe4a50a667a39a1ea6c23d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 707eaa96c0b24bd2be5f7310ca10c680 ({username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:56522)
I20260928 23:55:06.015666 26974 tablet_copy_service.cc:161] P 0591079a0cfe4a50a667a39a1ea6c23d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 707eaa96c0b24bd2be5f7310ca10c680 at {username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:56522: session id = 707eaa96c0b24bd2be5f7310ca10c680-00000000000000000000000000000000
I20260928 23:55:06.017172 26974 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:55:06.018759 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:55:06.020751 27418 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: tablet copy: Starting download of 0 data blocks...
I20260928 23:55:06.020955 27418 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: tablet copy: Starting download of 1 WAL segments...
I20260928 23:55:06.022581 27418 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:55:06.024302 26974 tablet_copy_service.cc:342] P 0591079a0cfe4a50a667a39a1ea6c23d: Request end of tablet copy session 707eaa96c0b24bd2be5f7310ca10c680-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:56522
I20260928 23:55:06.024468 26974 tablet_copy_service.cc:433] P 0591079a0cfe4a50a667a39a1ea6c23d: ending tablet copy session 707eaa96c0b24bd2be5f7310ca10c680-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 707eaa96c0b24bd2be5f7310ca10c680
I20260928 23:55:06.032389 27418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.034147 27532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.034142 27529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.036522 27530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.036801 27418 server_base.cc:1034] running on GCE node
I20260928 23:55:06.036978 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.037333 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.038488 27418 hybrid_clock.cc:648] HybridClock initialized: now 1790639706038490 us; error 48 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.040120 27418 init.cc:377] Logged in from keytab as oryx/127.20.88.61@KRBTEST.COM (short username oryx)
I20260928 23:55:06.040669 27418 webserver.cc:460] Webserver started at http://127.20.88.61:38509/ using document root <none> and password file <none>
I20260928 23:55:06.040872 27418 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.040935 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.042239 27418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:55:06.043248 27539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.043502 27418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:55:06.043599 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "707eaa96c0b24bd2be5f7310ca10c680"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "a7d3dbb043f5098faeb0a044f3f452f6"
server_key_iv: "349bf4b57f11842e0cc9334ff16aab04"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.043717 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.102656 27418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.105265 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.110071 27418 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.61:40149
I20260928 23:55:06.110116 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.61:40149 every 8 connection(s)
I20260928 23:55:06.110675 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
I20260928 23:55:06.112498 27592 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:55:06.113680 27592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Bootstrap starting.
I20260928 23:55:06.125025 27592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:55:06.125372 27592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Bootstrap complete.
I20260928 23:55:06.126000 27592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } }
I20260928 23:55:06.126140 27592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 707eaa96c0b24bd2be5f7310ca10c680, State: Initialized, Role: LEARNER
I20260928 23:55:06.126247 27592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } }
I20260928 23:55:06.126740 27592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:06.127323 27145 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
I20260928 23:55:06.128713 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } }
I20260928 23:55:06.128850 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:55:06.133566 27606 catalog_manager.cc:1259] Loaded cluster ID: 8d77d4c16fbe4e2e9d25b5f59a67af22
I20260928 23:55:06.133658 27606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: loading cluster ID for follower catalog manager: success
I20260928 23:55:06.135952 27606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: acquiring CA information for follower catalog manager: success
I20260928 23:55:06.137717 27606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:55:06.149641 27520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:06.149942 27520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:06.149986 27520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:06.154222 27520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:06.154302 27520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:06.154323 27520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:06.154340 27520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:06.154356 27520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:06.160120 27520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.60.keytab
--principal=oryx/127.20.88.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.88.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:06.161340 27520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:06.162380 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.171470 27614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.171456 27611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.171588 27612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.171769 27520 server_base.cc:1034] running on GCE node
I20260928 23:55:06.172688 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.173918 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.175115 27520 hybrid_clock.cc:648] HybridClock initialized: now 1790639706175096 us; error 32 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.178215 27520 init.cc:377] Logged in from keytab as oryx/127.20.88.60@KRBTEST.COM (short username oryx)
I20260928 23:55:06.179163 27520 webserver.cc:460] Webserver started at http://127.20.88.60:41871/ using document root <none> and password file <none>
I20260928 23:55:06.179725 27520 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.179788 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.179966 27520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:06.182199 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/instance:
uuid: "f485b0c71c404ef99a60244c59736e14"
format_stamp: "Formatted at 2026-09-28 23:55:06 on dist-test-slave-wvpw"
server_key: "f684418981a7e6cdf7e20b0b35fa8458"
server_key_iv: "13525427d3f64d9815cecae7fc0886a7"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.182761 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal/instance:
uuid: "f485b0c71c404ef99a60244c59736e14"
format_stamp: "Formatted at 2026-09-28 23:55:06 on dist-test-slave-wvpw"
server_key: "f684418981a7e6cdf7e20b0b35fa8458"
server_key_iv: "13525427d3f64d9815cecae7fc0886a7"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.186857 27520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:55:06.189700 27621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.191139 27520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:55:06.191316 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "f485b0c71c404ef99a60244c59736e14"
format_stamp: "Formatted at 2026-09-28 23:55:06 on dist-test-slave-wvpw"
server_key: "f684418981a7e6cdf7e20b0b35fa8458"
server_key_iv: "13525427d3f64d9815cecae7fc0886a7"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.191459 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.201201 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.204597 27520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:06.204826 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.212217 27520 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.60:36131
I20260928 23:55:06.212620 27673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.60:36131 every 8 connection(s)
I20260928 23:55:06.213349 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
I20260928 23:55:06.216383 27674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:06.218226 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27520
I20260928 23:55:06.220890 27674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.60@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:06.234861 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.223236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.20.88.62:41035)
0928 23:55:06.223377 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.223382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.223655 (+ 273us) server_negotiation.cc:407] Connection header received
0928 23:55:06.223790 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.223795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.223846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.223915 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.224812 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.225644 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.226989 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.227397 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.227484 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.230629 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.230633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.230636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.230676 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.232314 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.232862 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.232866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.232868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.232985 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.233505 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.233510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.233512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.233837 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.233938 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.234516 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.234688 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:55:06.235880 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.219876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.20.88.60:36131)
0928 23:55:06.220459 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.220476 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.220515 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:55:06.221216 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.221234 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.221651 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.222041 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.222929 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.224600 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.225902 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.226786 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.226967 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.229942 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.229949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.229969 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.229993 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.232270 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.232871 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.232878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.232879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.233059 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.233456 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.233459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.233461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.234143 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.234416 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.234651 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.234973 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":418,"thread_start_us":154,"threads_started":1}
I20260928 23:55:06.235882 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.223008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56530)
0928 23:55:06.223449 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.223501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.223691 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.223991 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.223997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.224473 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.224664 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.224674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.225790 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.225793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.226842 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.226851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.227613 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.227620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.227754 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.228420 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.228441 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.230400 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.232427 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.232436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.232449 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.232755 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.233182 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.233186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.233188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.233399 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.233958 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.233968 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.234211 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.235082 (+ 871us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.235282 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":1,"thread_start_us":156,"threads_started":1}
I20260928 23:55:06.237339 27674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.60@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
I20260928 23:55:06.246250 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.237804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48310)
0928 23:55:06.238202 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.238217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.238303 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.238564 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.238567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.238744 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.238852 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.238858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.239772 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.239779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.240423 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.240432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.240890 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.240894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.241005 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.241521 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.241535 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.242848 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.244777 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.244780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.244782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.244976 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.245282 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.245284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.245286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.245361 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.245790 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.245793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.245900 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.245996 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.246116 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260928 23:55:06.246320 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.237942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.20.88.61:40149)
0928 23:55:06.238203 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.238209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.238261 (+ 52us) server_negotiation.cc:407] Connection header received
0928 23:55:06.238399 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.238402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.238454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.238552 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.238973 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.239665 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.240546 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.240779 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.240911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.242987 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.242990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.242992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.243038 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.244678 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.245107 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.245112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.245114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.245206 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.245439 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.245441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.245443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.245677 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.245756 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.246055 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.246160 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":124,"threads_started":1}
I20260928 23:55:06.247133 27674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.60@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
I20260928 23:55:06.255565 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.247703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.20.88.60:36131)
0928 23:55:06.247821 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.247828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.247841 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:55:06.247971 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.247974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.248011 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.248105 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.248570 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.249418 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.250180 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.250424 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.250540 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.252513 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.252517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.252519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.252544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.253947 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.254357 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.254362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.254364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.254440 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.254722 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.254725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.254727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.254908 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.255017 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.255259 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.255416 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":56}
I20260928 23:55:06.255565 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.247626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:53998)
0928 23:55:06.247780 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.247794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.247870 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.248113 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.248116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.248352 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.248422 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.248427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.249554 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.249558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.250068 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.250075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.250533 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.250536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.250607 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.251079 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.251090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.252388 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.254064 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.254070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.254073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.254262 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.254547 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.254550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.254552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.254629 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.255004 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.255013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.255102 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.255206 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.255364 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:55:06.258989 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Bootstrap starting.
I20260928 23:55:06.261559 27674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:06.262364 27674 log.cc:824] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:06.264590 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: No bootstrap required, opened a new log
I20260928 23:55:06.267707 27674 raft_consensus.cc:348] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } }
I20260928 23:55:06.267956 27674 raft_consensus.cc:374] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:06.267992 27674 raft_consensus.cc:729] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f485b0c71c404ef99a60244c59736e14, State: Initialized, Role: FOLLOWER
I20260928 23:55:06.268646 27674 consensus_queue.cc:260] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } }
I20260928 23:55:06.269222 27680 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } } }
I20260928 23:55:06.269374 27680 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:06.270043 27674 sys_catalog.cc:564] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:55:06.275390 27691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:55:06.275497 27691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:55:06.278728 27520 master_runner.cc:428] Detected that this master f485b0c71c404ef99a60244c59736e14 is joining an existing cluster
I20260928 23:55:06.278805 27520 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.88.60:36131
I20260928 23:55:06.289548 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27016
I20260928 23:55:06.290977 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.281496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.20.88.60:36131)
0928 23:55:06.281627 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.281632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.281861 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:55:06.282036 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.282039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.282077 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.282125 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.282846 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.283651 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.284868 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.285222 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.285296 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.287149 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.287153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.287156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.287184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.288828 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.289465 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.289469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.289470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.289592 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.289883 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.289889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.289893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.290141 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.290223 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.290617 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.290791 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":50}
I20260928 23:55:06.291038 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.281476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:54006)
0928 23:55:06.281787 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.281801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.281923 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.282254 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.282257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.282591 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.282733 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.282741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.283773 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.283776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.284750 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.284760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.285472 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.285477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.285576 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.286624 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.286633 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.287004 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.288936 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.288940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.288941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.289209 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.289695 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.289701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.289706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.289798 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.290287 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.290293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.290400 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.290512 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.290717 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":26,"thread_start_us":116,"threads_started":1}
I20260928 23:55:06.297288 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:40145
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:41035
--webserver_port=44949
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:55:06.309865 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.281991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.20.88.61:40149)
0928 23:55:06.282136 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.282142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.282629 (+ 487us) server_negotiation.cc:407] Connection header received
0928 23:55:06.282802 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.282805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.282849 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.282906 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.283497 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.284278 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.285526 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.285895 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.286010 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.306059 (+ 20049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.306064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.306067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.306100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.307882 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308455 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.308458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.308460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.308556 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308851 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.308854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.308856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.309083 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.309152 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.309509 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.309669 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:55:06.310451 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.281469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.20.88.62:41035)
0928 23:55:06.281628 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.281634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.281983 (+ 349us) server_negotiation.cc:407] Connection header received
0928 23:55:06.282171 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.282177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.282230 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.282286 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.283077 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.283878 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.285998 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.286255 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.286317 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.306074 (+ 19757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.306081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.306086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.306128 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.308167 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308819 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.308826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.308830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.308929 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.309292 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.309298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.309303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.309590 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.309679 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.310134 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.310281 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:55:06.311125 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.282227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48320)
0928 23:55:06.282546 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.282561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.282693 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.283042 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.283048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.283257 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.283368 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.283375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.284408 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.284411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.285410 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.285419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.286132 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.286136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.286240 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.304947 (+ 18707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.304989 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.305815 (+ 826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.308039 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.308045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.308335 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.308668 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.308673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.308756 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.309222 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.309224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.309321 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.310713 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.310920 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":48,"spinlock_wait_cycles":8960,"thread_start_us":85,"threads_started":1}
I20260928 23:55:06.311159 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.281555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56532)
0928 23:55:06.281902 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.281916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.282052 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.282438 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.282442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.282790 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.282951 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.282958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.284006 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.284009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.285862 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.285874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.286442 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.286445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.286507 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.305004 (+ 18497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.305027 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.305816 (+ 789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.308368 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.308374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.308379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.308675 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.309063 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.309069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.309073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.309172 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.309765 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.309770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.309910 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.310662 (+ 752us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.310843 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":452,"spinlock_wait_cycles":1792,"thread_start_us":75,"threads_started":1}
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639704, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.62@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
I20260928 23:55:06.326301 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.316156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.20.88.60:36131)
0928 23:55:06.316389 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.316394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.316409 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:06.316593 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.316597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.316652 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.316726 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.317389 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.318202 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.319382 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.319758 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.319864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.322485 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.322489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.322492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.322522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.324449 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.324979 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.324983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.324985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.325079 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.325391 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.325394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.325396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.325629 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.325724 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.326023 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.326144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260928 23:55:06.326303 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.316079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:54010)
0928 23:55:06.316326 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.316342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.316432 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.316785 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.316789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.317049 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.317227 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.317239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.318328 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.318331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.319242 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.319254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.319891 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.319894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.320007 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.320613 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.320629 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.322307 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.324584 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.324588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.324590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.324869 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.325184 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.325187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.325190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.325293 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.325727 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.325730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.325848 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.325957 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.326120 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 23:55:06.327730 26954 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.88.60:36131: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f485b0c71c404ef99a60244c59736e14"
member_type: NON_VOTER
last_known_addr {
host: "127.20.88.60"
port: 36131
}
attrs {
promote: true
}
}
dest_uuid: "0591079a0cfe4a50a667a39a1ea6c23d"
cas_config_opid_index: 5
I20260928 23:55:06.328400 26954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } }
I20260928 23:55:06.329519 27515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f485b0c71c404ef99a60244c59736e14 (127.20.88.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.330729 27648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:06.331000 27515 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.88.60:36131
I20260928 23:55:06.331190 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.331279 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:06.332150 27648 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:55:06.332197 27520 master.cc:537] Master@127.20.88.60:36131 shutting down...
I20260928 23:55:06.333097 27517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:55:06.334856 26941 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d -> Peer f485b0c71c404ef99a60244c59736e14 (127.20.88.60:36131): Couldn't send request to peer f485b0c71c404ef99a60244c59736e14. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:55:06.339766 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.330010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.20.88.61:40149)
0928 23:55:06.333112 (+ 3102us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.333118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.333277 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:55:06.333463 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.333467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.333539 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.333604 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:06.335102 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.335918 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.338058 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.338739 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.338819 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.338975 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.339533 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.339621 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3005}
I20260928 23:55:06.340329 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.329764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:48332)
0928 23:55:06.333212 (+ 3448us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.333227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.333348 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.333758 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.333761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.333779 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:06.334964 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.334973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.336123 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.336129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.337779 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.337792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.339246 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.339250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.339346 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.339400 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.340083 (+ 683us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.340171 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3370}
I20260928 23:55:06.340823 27520 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:06.340859 27566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 LEARNER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:55:06.341117 27520 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:06.341212 27520 tablet_replica.cc:330] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: stopping tablet replica
I20260928 23:55:06.341275 27517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 23:55:06.343168 27566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f485b0c71c404ef99a60244c59736e14 (127.20.88.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.347484 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0591079a0cfe4a50a667a39a1ea6c23d. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.347630 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:55:06.347723 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: NON_VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: true } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.347786 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:55:06.348523 27517 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: attempting to promote NON_VOTER 707eaa96c0b24bd2be5f7310ca10c680 to VOTER
I20260928 23:55:06.348989 27517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } }
I20260928 23:55:06.350148 27565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 LEARNER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 23:55:06.350777 27515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 23:55:06.351140 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.350670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:54016)
0928 23:55:06.350975 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.351049 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:36131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260928 23:55:06.351848 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.351614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:54028)
0928 23:55:06.351751 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.351793 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:36131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 23:55:06.351933 26941 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d -> Peer f485b0c71c404ef99a60244c59736e14 (127.20.88.60:36131): Couldn't send request to peer f485b0c71c404ef99a60244c59736e14. Status: Network error: Client connection negotiation failed: client connection to 127.20.88.60:36131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:55:06.352924 27516 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 707eaa96c0b24bd2be5f7310ca10c680 (127.20.88.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.353461 27566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 707eaa96c0b24bd2be5f7310ca10c680 (127.20.88.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.354457 27516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.354571 27516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:06.354626 27517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.354730 27517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:06.355707 27702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.355825 27702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:06.356758 27279 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
I20260928 23:55:06.357659 27520 master.cc:552] Master@127.20.88.60:36131 shutdown complete.
I20260928 23:55:06.357704 27520 master_runner.cc:305] Clearing existing system tablet
I20260928 23:55:06.357950 27520 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.358006 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.359332 27520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 23:55:06.360318 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.360548 27520 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.360675 27520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:06.360707 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "f485b0c71c404ef99a60244c59736e14"
format_stamp: "Formatted at 2026-09-28 23:55:06 on dist-test-slave-wvpw"
server_key: "f684418981a7e6cdf7e20b0b35fa8458"
server_key_iv: "13525427d3f64d9815cecae7fc0886a7"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.362859 27520 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:06.365772 27520 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:55:06.365880 27520 log.cc:1196] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 23:55:06.366127 27520 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Deleting consensus metadata
I20260928 23:55:06.366345 27520 master_runner.cc:315] Copying system tablet from 127.20.88.62:41035
I20260928 23:55:06.367532 27520 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: tablet copy: Beginning tablet copy session from remote peer at address 127.20.88.62:41035
I20260928 23:55:06.377894 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.368101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:56548)
0928 23:55:06.368434 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.368449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.368577 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.369011 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.369013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.369234 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.369365 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.369373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.370450 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.370454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.371618 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.371631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.372335 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.372339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.372437 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.373532 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.373548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.373986 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.376129 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.376133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.376135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.376424 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.376735 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.376738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.376740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.376834 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.377289 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.377292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.377402 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.377511 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.377670 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
I20260928 23:55:06.377907 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.368208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.20.88.62:41035)
0928 23:55:06.368383 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.368388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.368537 (+ 149us) server_negotiation.cc:407] Connection header received
0928 23:55:06.368742 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.368746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.368803 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.368858 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.369507 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.370330 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.371765 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.372153 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.372354 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.374175 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.374179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.374182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.374214 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.375991 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.376535 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.376539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.376541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.376634 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.376939 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.376942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.376944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.377187 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.377282 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.377609 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.377744 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 23:55:06.378775 26974 tablet_copy_service.cc:140] P 0591079a0cfe4a50a667a39a1ea6c23d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f485b0c71c404ef99a60244c59736e14 ({username='oryx', principal='oryx/127.20.88.60@KRBTEST.COM'} at 127.0.0.1:56548)
I20260928 23:55:06.378923 26974 tablet_copy_service.cc:161] P 0591079a0cfe4a50a667a39a1ea6c23d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f485b0c71c404ef99a60244c59736e14 at {username='oryx', principal='oryx/127.20.88.60@KRBTEST.COM'} at 127.0.0.1:56548: session id = f485b0c71c404ef99a60244c59736e14-00000000000000000000000000000000
I20260928 23:55:06.380092 26974 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:55:06.381556 27520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:55:06.384126 27520 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: tablet copy: Starting download of 0 data blocks...
I20260928 23:55:06.384436 27520 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: tablet copy: Starting download of 1 WAL segments...
I20260928 23:55:06.386186 27520 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:55:06.388132 26974 tablet_copy_service.cc:342] P 0591079a0cfe4a50a667a39a1ea6c23d: Request end of tablet copy session f485b0c71c404ef99a60244c59736e14-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.88.60@KRBTEST.COM'} at 127.0.0.1:56548
I20260928 23:55:06.388303 26974 tablet_copy_service.cc:433] P 0591079a0cfe4a50a667a39a1ea6c23d: ending tablet copy session f485b0c71c404ef99a60244c59736e14-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f485b0c71c404ef99a60244c59736e14
I20260928 23:55:06.397166 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.399211 27712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.399211 27715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.399624 27520 server_base.cc:1034] running on GCE node
W20260928 23:55:06.400404 27713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.400776 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.401108 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.402256 27520 hybrid_clock.cc:648] HybridClock initialized: now 1790639706402244 us; error 32 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.404127 27520 init.cc:377] Logged in from keytab as oryx/127.20.88.60@KRBTEST.COM (short username oryx)
I20260928 23:55:06.404727 27520 webserver.cc:460] Webserver started at http://127.20.88.60:34035/ using document root <none> and password file <none>
I20260928 23:55:06.404929 27520 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.404968 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.406747 27520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:55:06.407867 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.408169 27520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:55:06.408318 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "f485b0c71c404ef99a60244c59736e14"
format_stamp: "Formatted at 2026-09-28 23:55:06 on dist-test-slave-wvpw"
server_key: "f684418981a7e6cdf7e20b0b35fa8458"
server_key_iv: "13525427d3f64d9815cecae7fc0886a7"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.408406 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.418349 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.420825 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.426167 27520 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.60:36131
I20260928 23:55:06.426190 27774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.60:36131 every 8 connection(s)
I20260928 23:55:06.427017 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
I20260928 23:55:06.430136 27775 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:55:06.431303 27775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Bootstrap starting.
I20260928 23:55:06.444370 27775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:55:06.444741 27775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: Bootstrap complete.
I20260928 23:55:06.445434 27775 raft_consensus.cc:348] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } }
I20260928 23:55:06.445595 27775 raft_consensus.cc:729] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f485b0c71c404ef99a60244c59736e14, State: Initialized, Role: LEARNER
I20260928 23:55:06.445712 27775 consensus_queue.cc:260] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } }
I20260928 23:55:06.446108 27778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.446230 27775 sys_catalog.cc:564] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:06.446327 27778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:55:06.450666 27789 catalog_manager.cc:1259] Loaded cluster ID: 8d77d4c16fbe4e2e9d25b5f59a67af22
I20260928 23:55:06.450838 27789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: loading cluster ID for follower catalog manager: success
I20260928 23:55:06.453147 27789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: acquiring CA information for follower catalog manager: success
I20260928 23:55:06.454726 27789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:55:06.458165 27700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:06.458446 27700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:06.458526 27700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:06.462904 27700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:06.462983 27700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:06.463085 27700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:06.468842 27700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:40145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=44949
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:06.470129 27700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:06.471267 27700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.482148 27795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.482137 27794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.482345 27700 server_base.cc:1034] running on GCE node
W20260928 23:55:06.482137 27797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.483007 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.483698 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.484927 27700 hybrid_clock.cc:648] HybridClock initialized: now 1790639706484875 us; error 66 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.488065 27700 init.cc:377] Logged in from keytab as oryx/127.20.88.1@KRBTEST.COM (short username oryx)
I20260928 23:55:06.489073 27700 webserver.cc:460] Webserver started at http://127.20.88.1:44949/ using document root <none> and password file <none>
I20260928 23:55:06.489655 27700 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.489741 27700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.494004 27700 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:55:06.496173 27804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.497699 27700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:55:06.497843 27700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "56800e01ac454625836872974887c3e3"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "b8d51202e8d037dd40d01a95b300af71"
server_key_iv: "5a4a70f7fa738470a4531bd44d039f9a"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.498440 27700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.520109 27700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.523624 27700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:06.523861 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.524641 27700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:06.525573 27700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:06.525620 27700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.525692 27700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:06.525722 27700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.535027 27700 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:40145
I20260928 23:55:06.535075 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:40145 every 8 connection(s)
I20260928 23:55:06.536217 27700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:06.538472 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27700
I20260928 23:55:06.538699 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27150
I20260928 23:55:06.546878 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:42383
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:41035
--webserver_port=41087
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
I20260928 23:55:06.557322 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.540095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:44191 (local address 127.20.88.62:41035)
0928 23:55:06.540251 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.540258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.542030 (+ 1772us) server_negotiation.cc:407] Connection header received
0928 23:55:06.542209 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.542212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.542272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.542326 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.543858 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.544755 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.546870 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547227 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547315 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.552031 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.552038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.552044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.552086 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.554013 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.554870 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.554877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.554881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.554985 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555505 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.555512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.555515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.555789 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.555898 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.556891 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.557052 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:55:06.557523 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.538142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:39131 (local address 127.20.88.61:40149)
0928 23:55:06.539391 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.539396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.539911 (+ 515us) server_negotiation.cc:407] Connection header received
0928 23:55:06.541404 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.541408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.541475 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.541545 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.543922 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.544759 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.546876 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547253 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547324 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.551859 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.551864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.551870 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.551909 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.554671 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555162 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.555166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.555168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.555252 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555559 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.555563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.555564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.555758 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.555835 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.557255 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.557376 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149}
I20260928 23:55:06.558314 27923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.538726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.20.88.1:39131)
0928 23:55:06.539730 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.539796 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.541124 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.542009 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.542020 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.542732 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.543675 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.543687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.545029 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.545031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.546722 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.546733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547961 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.548065 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.549023 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.549043 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.551595 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.554809 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.554811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.554814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.555046 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555373 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.555378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.555461 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555916 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.555918 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.556367 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.557916 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.558062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":344,"spinlock_wait_cycles":4224,"thread_start_us":133,"threads_started":1}
I20260928 23:55:06.558648 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.538726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.20.88.1:35797)
0928 23:55:06.541557 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.541572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.541669 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.542098 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.542104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.542683 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.543674 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.543689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.545352 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.545359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.546713 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.546724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547583 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.547853 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.548819 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.548850 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.552222 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.554506 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.554510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.554513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.554823 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555143 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.555148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.555379 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555909 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.555916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.556369 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.556614 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.558486 (+ 1872us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2728,"mutex_wait_us":1002,"spinlock_wait_cycles":66176,"thread_start_us":108,"threads_started":1}
I20260928 23:55:06.558314 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.539882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:44191)
0928 23:55:06.541938 (+ 2056us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.541950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.542090 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.542485 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.542490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.542764 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.543664 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.543687 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.544996 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.544999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.546722 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.546733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547574 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.547839 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.548871 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.548882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.551843 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.554180 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.554190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.554214 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.554727 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555109 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.555120 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.555362 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.555950 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.555953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.556377 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.556614 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.557121 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"mutex_wait_us":532,"spinlock_wait_cycles":17664,"thread_start_us":89,"threads_started":1}
I20260928 23:55:06.559417 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.538217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:35797 (local address 127.20.88.60:36131)
0928 23:55:06.539876 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.539881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.541644 (+ 1763us) server_negotiation.cc:407] Connection header received
0928 23:55:06.541772 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.541778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.541846 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.541941 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.543857 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.545209 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.546865 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.547291 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.547415 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.552436 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.552441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.552448 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.552486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.554358 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.554939 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.554945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.554949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.555038 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.555486 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.555493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.555498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.555760 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.555849 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.559073 (+ 3224us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.559232 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1578,"thread_start_us":86,"threads_started":1}
I20260928 23:55:06.560269 27920 heartbeater.cc:347] Connected to a master server at 127.20.88.61:40149
I20260928 23:55:06.560319 27918 heartbeater.cc:347] Connected to a master server at 127.20.88.60:36131
I20260928 23:55:06.560385 27919 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:06.560537 27919 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.560537 27920 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.560536 27918 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.561192 27919 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
I20260928 23:55:06.561192 27920 heartbeater.cc:511] Master 127.20.88.61:40149 requested a full tablet report, sending...
I20260928 23:55:06.561215 27918 heartbeater.cc:511] Master 127.20.88.60:36131 requested a full tablet report, sending...
I20260928 23:55:06.562784 27555 ts_manager.cc:194] Registered new tserver with Master: 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:06.562909 27739 ts_manager.cc:194] Registered new tserver with Master: 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:06.563591 26954 ts_manager.cc:194] Re-registered known tserver with Master: 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:06.564209 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:44191
I20260928 23:55:06.661239 27413 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
W20260928 23:55:06.673590 27926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:06.673962 27926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:06.674052 27926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:06.678300 27926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:06.678406 27926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:06.678516 27926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:06.684026 27926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:42383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=41087
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:06.685268 27926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:06.686355 27926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.696492 27931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.696521 27932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.696494 27934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.697038 27926 server_base.cc:1034] running on GCE node
I20260928 23:55:06.697490 27926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.698200 27926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.699404 27926 hybrid_clock.cc:648] HybridClock initialized: now 1790639706699369 us; error 57 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.702627 27926 init.cc:377] Logged in from keytab as oryx/127.20.88.2@KRBTEST.COM (short username oryx)
I20260928 23:55:06.703634 27926 webserver.cc:460] Webserver started at http://127.20.88.2:41087/ using document root <none> and password file <none>
I20260928 23:55:06.704303 27926 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.704396 27926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.708676 27926 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:55:06.711040 27941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.712663 27926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:06.712815 27926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "e6528b323b0540a8b6ce27d2068abce9"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "032bc4a52459a6dd958f0aa7d3895914"
server_key_iv: "6f0e8c6c46b1442648fac32484883344"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.713446 27926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.733333 27926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.737104 27926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:06.737313 27926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.737952 27926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:06.738847 27926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:06.738904 27926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.738947 27926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:06.738999 27926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.748927 27926 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:42383
I20260928 23:55:06.748991 28054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:42383 every 8 connection(s)
I20260928 23:55:06.750242 27926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:06.755394 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 27926
I20260928 23:55:06.755569 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27284
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
I20260928 23:55:06.765455 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:43281
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:41035
--webserver_port=38181
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:55:06.770152 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.752412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50189 (local address 127.20.88.60:36131)
0928 23:55:06.752581 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.752587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.753522 (+ 935us) server_negotiation.cc:407] Connection header received
0928 23:55:06.754685 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.754690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.754745 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.754798 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.756649 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757489 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.759920 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760305 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760384 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.764867 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.764872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.764875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.764913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.767271 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768076 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.768079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.768081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.768172 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768577 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.768581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.768582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.768839 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.768930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.769817 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.769967 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:55:06.770172 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.752314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:39751 (local address 127.20.88.61:40149)
0928 23:55:06.752456 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.752460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.753534 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 23:55:06.754674 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.754678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.754733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.754839 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.756649 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757551 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.759913 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760376 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.765028 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.765033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.765038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.765069 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.767460 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768092 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.768100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.768103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.768193 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768605 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.768611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.768613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.768878 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.768970 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.769843 (+ 873us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.769984 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:55:06.771147 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.752853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.20.88.2:50189)
0928 23:55:06.753355 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.753411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.754476 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.755022 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.755031 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.755642 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.756468 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.756490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757685 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.757689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.759756 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.759769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760618 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.760928 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.761980 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.761992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.764616 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.767416 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.767424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.767444 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.767936 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.768309 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.768314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.768469 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.768981 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.768988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.769403 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.770380 (+ 977us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.770650 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":934,"spinlock_wait_cycles":79872,"thread_start_us":143,"threads_started":1}
I20260928 23:55:06.771143 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.752852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.20.88.2:39751)
0928 23:55:06.753355 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.753411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.754461 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.755022 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.755029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.755697 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.756480 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.756493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757702 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.757708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.759764 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.759778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760674 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.760917 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.762146 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.762167 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.764867 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.767566 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.767570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.767572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.767936 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.768344 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.768351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.768483 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.769048 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.769053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.769406 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.770330 (+ 924us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.770553 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":446,"spinlock_wait_cycles":83456,"thread_start_us":153,"threads_started":1}
I20260928 23:55:06.771545 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.754093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:44155 (local address 127.20.88.62:41035)
0928 23:55:06.754223 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.754228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.754296 (+ 68us) server_negotiation.cc:407] Connection header received
0928 23:55:06.754692 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.754696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.754744 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.754798 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.756648 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757492 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.759914 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760298 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.764866 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.764873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.764879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.764916 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.767268 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768690 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.768693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.768695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.768781 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.769105 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.769108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.769110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.769319 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.769411 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.771270 (+ 1859us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.771400 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:55:06.771144 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.753936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:44155)
0928 23:55:06.754221 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.754236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.754476 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.755022 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.755031 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.756170 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.756468 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.756490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.757685 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.757689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.759764 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.759769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.760618 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.760623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.760929 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.761942 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.761968 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.764699 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.767438 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.767441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.767451 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.768576 (+ 1125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.768898 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.768903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.768905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.768987 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.769454 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.769457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.769565 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.769739 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.770113 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":701,"spinlock_wait_cycles":11648,"thread_start_us":103,"threads_started":1}
I20260928 23:55:06.772543 28055 heartbeater.cc:347] Connected to a master server at 127.20.88.60:36131
I20260928 23:55:06.772614 28057 heartbeater.cc:347] Connected to a master server at 127.20.88.61:40149
I20260928 23:55:06.772794 28057 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.772794 28055 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.772614 28056 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:06.773320 28056 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.773417 28055 heartbeater.cc:511] Master 127.20.88.60:36131 requested a full tablet report, sending...
I20260928 23:55:06.773461 28056 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
I20260928 23:55:06.773416 28057 heartbeater.cc:511] Master 127.20.88.61:40149 requested a full tablet report, sending...
I20260928 23:55:06.774724 27555 ts_manager.cc:194] Registered new tserver with Master: e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383)
I20260928 23:55:06.774818 27739 ts_manager.cc:194] Registered new tserver with Master: e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383)
I20260928 23:55:06.775377 26954 ts_manager.cc:194] Re-registered known tserver with Master: e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383)
I20260928 23:55:06.776145 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:44155
I20260928 23:55:06.852020 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.847873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:54038)
0928 23:55:06.848131 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.848144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.848301 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.848624 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.848626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.848642 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:06.848742 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.848746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.849839 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.849842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.850655 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.850665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.851607 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.851611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.851729 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.851751 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.851815 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.851885 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 23:55:06.852039 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.847947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.20.88.60:36131)
0928 23:55:06.848101 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.848106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.848262 (+ 156us) server_negotiation.cc:407] Connection header received
0928 23:55:06.848423 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.848426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.848492 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.848610 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:06.848849 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.849712 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.850787 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.851452 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.851564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.851731 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.851832 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.851907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:55:06.852916 27749 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 23:55:06.853369 27778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0591079a0cfe4a50a667a39a1ea6c23d. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:06.853529 27778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: LEARNER
W20260928 23:55:06.893306 28062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:06.893611 28062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:06.893694 28062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:06.897835 28062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:06.897939 28062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:06.898036 28062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:06.903930 28062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:43281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=38181
--tserver_master_addrs=127.20.88.62:41035,127.20.88.61:40149,127.20.88.60:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:06.905144 28062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:06.906275 28062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:06.916945 28068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.917166 28062 server_base.cc:1034] running on GCE node
W20260928 23:55:06.917205 28067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:06.917343 28070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:06.917912 28062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:06.918807 28062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:06.920032 28062 hybrid_clock.cc:648] HybridClock initialized: now 1790639706919989 us; error 59 us; skew 500 ppm
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:06.923166 28062 init.cc:377] Logged in from keytab as oryx/127.20.88.3@KRBTEST.COM (short username oryx)
I20260928 23:55:06.924149 28062 webserver.cc:460] Webserver started at http://127.20.88.3:38181/ using document root <none> and password file <none>
I20260928 23:55:06.924814 28062 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:06.924973 28062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:06.929587 28062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:55:06.931970 28077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.933646 28062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:55:06.933811 28062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
format_stamp: "Formatted at 2026-09-28 23:55:05 on dist-test-slave-wvpw"
server_key: "bc8bf8e430a3be35ebf8bf7e5a511aa3"
server_key_iv: "6625d6c0467029d98fe26dc2145ebf11"
server_key_version: "encryptionkey@0"
I20260928 23:55:06.934446 28062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:06.954627 28062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:06.958161 28062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:06.958387 28062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:06.959051 28062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:06.959995 28062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:06.960068 28062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.960142 28062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:06.960192 28062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:06.969662 28062 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:43281
I20260928 23:55:06.969699 28190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:43281 every 8 connection(s)
I20260928 23:55:06.970896 28062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:06.977898 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 28062
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.60@KRBTEST.COM
Sep 28 23:55:06 dist-test-slave-wvpw krb5kdc[26909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639706, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.61@KRBTEST.COM
I20260928 23:55:06.990657 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.972898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:38763 (local address 127.20.88.60:36131)
0928 23:55:06.973027 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.973031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.974334 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 23:55:06.975647 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.975651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.975731 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.975791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.977419 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978246 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980409 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980750 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980826 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.985184 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.985188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.985191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.985220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.987889 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.988628 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.988633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.988636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.988735 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.989179 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.989183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.989185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.989422 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.989519 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.990316 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.990458 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:55:06.991536 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.973051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:34339 (local address 127.20.88.61:40149)
0928 23:55:06.973156 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.973161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.974347 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 23:55:06.975739 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.975743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.975799 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.975866 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.977419 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978226 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980186 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980607 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980719 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.985438 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.985442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.985445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.985471 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.987392 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.989546 (+ 2154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.989550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.989551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.989639 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.989935 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.989941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.989946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.990142 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.990222 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.991221 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.991357 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:55:06.992589 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.973510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.20.88.3:34339)
0928 23:55:06.974136 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.974211 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.975401 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.976009 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.976017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.976563 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.977262 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.977271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978363 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.978369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.980034 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.980044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980898 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.981156 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.982151 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.982178 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.985284 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.988899 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.988904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.988907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.989422 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.989741 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.989745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.989747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.989828 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.990806 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.990809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.990936 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.991037 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.991327 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":472,"spinlock_wait_cycles":103808,"thread_start_us":133,"threads_started":1}
I20260928 23:55:06.992937 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.974707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:39955)
0928 23:55:06.974984 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.974998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.975420 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.976008 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.976016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.976553 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.977249 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.977265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978312 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.978317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.980214 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.980221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980947 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.981217 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.982284 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.982300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.985991 (+ 3691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.988956 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.988959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.988961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.989211 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.990261 (+ 1050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.990264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.990266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.990345 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.991769 (+ 1424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.991773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.991889 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.992003 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.992157 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":646,"spinlock_wait_cycles":12160,"thread_start_us":79,"threads_started":1}
I20260928 23:55:06.993595 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.974809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:39955 (local address 127.20.88.62:41035)
0928 23:55:06.974946 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.974949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.975061 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:55:06.975632 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.975636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.975695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.975803 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.977402 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978130 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980360 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980697 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.986467 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.986471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.986474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.986503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.988841 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.989339 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.989343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.989345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.989430 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.990451 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.990454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.990456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.990679 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.990765 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.993254 (+ 2489us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.993423 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":54}
I20260928 23:55:06.993675 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.973510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.20.88.3:38763)
0928 23:55:06.974136 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:06.974211 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:06.975401 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:06.976009 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:06.976017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:06.976557 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:06.977262 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.977271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.978502 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.978506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.980280 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:06.980289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.980996 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.980999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:06.981185 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.982831 (+ 1646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:06.982848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:06.984979 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:06.988031 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.988038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.988059 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.988499 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.988855 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:06.988859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:06.988863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:06.989065 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:06.989582 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:06.989591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:06.989989 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 23:55:06.990906 (+ 917us) client_negotiation.cc:240] Negotiation successful
0928 23:55:06.993492 (+ 2586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":761,"spinlock_wait_cycles":21888,"thread_start_us":151,"threads_started":1}
I20260928 23:55:06.994156 28192 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41035
I20260928 23:55:06.994246 28193 heartbeater.cc:347] Connected to a master server at 127.20.88.61:40149
I20260928 23:55:06.994422 28193 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.994419 28192 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.994678 28191 heartbeater.cc:347] Connected to a master server at 127.20.88.60:36131
I20260928 23:55:06.994771 28191 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:06.995061 28193 heartbeater.cc:511] Master 127.20.88.61:40149 requested a full tablet report, sending...
I20260928 23:55:06.995078 28192 heartbeater.cc:511] Master 127.20.88.62:41035 requested a full tablet report, sending...
I20260928 23:55:06.995061 28191 heartbeater.cc:511] Master 127.20.88.60:36131 requested a full tablet report, sending...
I20260928 23:55:06.995270 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.980178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.20.88.62:41035)
0928 23:55:06.982424 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.982435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.982450 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:06.982491 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.982494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.982546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.982639 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.983620 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.984370 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.985611 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.986040 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.986148 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.987986 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.987990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.987993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.988023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.990618 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.992192 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.992196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.992198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.992319 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.992617 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.992621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.992623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.993375 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.993497 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.994951 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.995097 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2149,"thread_start_us":106,"threads_started":1}
I20260928 23:55:06.996398 27555 ts_manager.cc:194] Registered new tserver with Master: 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
I20260928 23:55:06.996484 27739 ts_manager.cc:194] Registered new tserver with Master: 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
I20260928 23:55:06.997308 26954 ts_manager.cc:194] Re-registered known tserver with Master: 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
I20260928 23:55:06.998075 26954 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:39955
I20260928 23:55:07.000110 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.981341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.20.88.61:40149)
0928 23:55:06.982250 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.982256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.983829 (+ 1573us) server_negotiation.cc:407] Connection header received
0928 23:55:06.983873 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.983878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.983935 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.984029 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.986462 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.987291 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.989546 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.989907 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.990002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.992308 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.992312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.992317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.992346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.996537 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.997992 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.997996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.997998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.998090 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:06.998418 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:06.998425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:06.998428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:06.998767 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:06.998902 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:06.999578 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 23:55:06.999713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":95,"threads_started":1}
I20260928 23:55:07.001890 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:06.980685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.20.88.60:36131)
0928 23:55:06.983178 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:06.983182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:06.983195 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:55:06.983230 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:06.983234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:06.983290 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:06.983378 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:06.984955 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.985729 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.990095 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:06.990479 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:06.990569 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:06.997662 (+ 7093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:06.997665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:06.997667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:06.997690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:06.999961 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.000480 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.000484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.000486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.000570 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.000877 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.000880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.000882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.001092 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:07.001216 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.001485 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.001725 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2415,"thread_start_us":94,"threads_started":1}
I20260928 23:55:07.000494 26954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:07.004525 26954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:07.007014 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: NON_VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: true } } }
I20260928 23:55:07.007164 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:07.010594 28211 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: attempting to promote NON_VOTER f485b0c71c404ef99a60244c59736e14 to VOTER
I20260928 23:55:07.010955 28211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } }
I20260928 23:55:07.012290 27749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 LEARNER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 23:55:07.012744 27515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:55:07.013144 27566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Refusing update from remote peer 0591079a0cfe4a50a667a39a1ea6c23d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 23:55:07.013561 27515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:55:07.014434 28210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f485b0c71c404ef99a60244c59736e14 (127.20.88.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.015563 27566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f485b0c71c404ef99a60244c59736e14 (127.20.88.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.016142 28209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.016296 28209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:07.016422 28209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.016160 28210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.016649 28210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:07.016495 28209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:07.017156 27749 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f485b0c71c404ef99a60244c59736e14 (127.20.88.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.018018 28219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.018157 28219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:07.019253 27778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:07.019443 27778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:07.029366 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.022165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43281 (local address 127.0.0.1:46854)
0928 23:55:07.022322 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:07.022339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:07.022435 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:07.024906 (+ 2471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:07.024909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:07.024938 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:07.025052 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.025058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.026399 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.026402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.027610 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.027622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.028918 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.028922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.029012 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.029035 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:55:07.029100 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:55:07.029195 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":23,"spinlock_wait_cycles":1408}
I20260928 23:55:07.029415 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.022615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.20.88.3:43281)
0928 23:55:07.024458 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.024472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.024497 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:55:07.024568 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.024573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.024755 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.025190 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:07.025318 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.026273 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.027774 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.028768 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.028890 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.029063 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.029130 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.029241 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1701,"thread_start_us":105,"threads_started":1}
I20260928 23:55:07.029991 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.022472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:40145 (local address 127.0.0.1:43954)
0928 23:55:07.022775 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:07.022789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:07.022882 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:07.023951 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:07.023955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:07.023979 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:07.024087 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.024092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.025447 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.025451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.028043 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.028050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.029175 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.029178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.029256 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.029277 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:55:07.029339 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:55:07.029418 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":23,"thread_start_us":143,"threads_started":1}
I20260928 23:55:07.030157 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.022753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:42383 (local address 127.0.0.1:34648)
0928 23:55:07.024048 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:07.024062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:07.024146 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:07.024866 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:07.024869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:07.024888 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:07.025043 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.025051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.026357 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.026360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.027610 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.027622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.029696 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.029702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.029798 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.029822 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:55:07.029879 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:55:07.029947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":36,"thread_start_us":73,"threads_started":1}
I20260928 23:55:07.030329 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.023144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.20.88.2:42383)
0928 23:55:07.024470 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.024479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.024497 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:55:07.024561 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.024565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.024721 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.024886 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:07.025171 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.026223 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.027774 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.028768 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.028890 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.030038 (+ 1148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.030095 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.030188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":110,"threads_started":1}
I20260928 23:55:07.030977 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.20.88.1:40145)
0928 23:55:07.023349 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.023369 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.023395 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:55:07.023482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.023487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.023762 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.023960 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:07.024180 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.025331 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.028166 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.028812 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.028950 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.029117 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.030660 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.030821 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":108,"threads_started":1}
I20260928 23:55:07.032703 28125 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c (DEFAULT_TABLE table=test-table [id=1f5a38461886422584976775c98804bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:07.033519 27852 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c (DEFAULT_TABLE table=test-table [id=1f5a38461886422584976775c98804bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:07.033957 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9ad3ca4da645d19bc47dd9e152424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:07.034426 27852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9ad3ca4da645d19bc47dd9e152424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:07.034307 27989 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c (DEFAULT_TABLE table=test-table [id=1f5a38461886422584976775c98804bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:07.035281 27989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9ad3ca4da645d19bc47dd9e152424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:07.043246 28226 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Bootstrap starting.
I20260928 23:55:07.044118 28227 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Bootstrap starting.
I20260928 23:55:07.045797 28226 tablet_bootstrap.cc:654] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:07.046278 28227 tablet_bootstrap.cc:654] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:07.046316 28225 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Bootstrap starting.
I20260928 23:55:07.046834 28226 log.cc:824] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:07.047168 28227 log.cc:824] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:07.048763 28225 tablet_bootstrap.cc:654] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:07.049301 28226 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: No bootstrap required, opened a new log
I20260928 23:55:07.049301 28227 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: No bootstrap required, opened a new log
I20260928 23:55:07.049521 28226 ts_tablet_manager.cc:1397] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:55:07.049520 28227 ts_tablet_manager.cc:1397] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:55:07.049621 28225 log.cc:824] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:07.051486 28225 tablet_bootstrap.cc:492] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: No bootstrap required, opened a new log
I20260928 23:55:07.051716 28225 ts_tablet_manager.cc:1397] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:55:07.054534 28226 raft_consensus.cc:348] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.054534 28227 raft_consensus.cc:348] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.054877 28226 raft_consensus.cc:374] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:07.054877 28227 raft_consensus.cc:374] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:07.054930 28226 raft_consensus.cc:729] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aded145201948e8a1af2d4a2bc1e3d1, State: Initialized, Role: FOLLOWER
I20260928 23:55:07.054930 28227 raft_consensus.cc:729] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6528b323b0540a8b6ce27d2068abce9, State: Initialized, Role: FOLLOWER
I20260928 23:55:07.055462 28227 consensus_queue.cc:260] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.055462 28226 consensus_queue.cc:260] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.055719 28225 raft_consensus.cc:348] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.056021 28225 raft_consensus.cc:374] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:07.056083 28225 raft_consensus.cc:729] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56800e01ac454625836872974887c3e3, State: Initialized, Role: FOLLOWER
I20260928 23:55:07.056674 28226 ts_tablet_manager.cc:1428] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:55:07.056674 28227 ts_tablet_manager.cc:1428] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 23:55:07.057109 28056 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
I20260928 23:55:07.058046 28192 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
I20260928 23:55:07.057827 28225 consensus_queue.cc:260] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.060047 28225 ts_tablet_manager.cc:1428] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 23:55:07.060504 27919 heartbeater.cc:503] Master 127.20.88.62:41035 was elected leader, sending a full tablet report...
I20260928 23:55:07.089303 28231 raft_consensus.cc:482] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:07.089617 28231 raft_consensus.cc:504] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.090958 28231 leader_election.cc:288] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281), 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:07.095793 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.091394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38331 (local address 127.20.88.3:43281)
0928 23:55:07.091628 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.091633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.091649 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:07.091714 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.091717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.091768 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.091853 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:07.092263 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.092842 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.094751 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.095184 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.095275 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.095332 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.095581 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.095660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 23:55:07.095793 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.091175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43281 (local address 127.20.88.2:38331)
0928 23:55:07.091324 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:07.091342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:07.091451 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:07.091926 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:07.091929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:07.091948 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:07.092105 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.092113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.092977 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.092983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.094593 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.094604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.095313 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.095316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.095428 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.095452 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:55:07.095548 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:55:07.095640 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":190}
I20260928 23:55:07.096542 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.091196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:40145 (local address 127.20.88.2:43173)
0928 23:55:07.091644 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:07.091657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:07.091752 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:07.092062 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:07.092065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:07.092079 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:07.092194 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.092201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.092995 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.092998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.094590 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:07.094604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.096115 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.096121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:07.096224 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.096269 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:55:07.096345 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:55:07.096421 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":483}
I20260928 23:55:07.096573 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.091355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:43173 (local address 127.20.88.1:40145)
0928 23:55:07.091516 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.091529 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.091771 (+ 242us) server_negotiation.cc:407] Connection header received
0928 23:55:07.091891 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.091895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.091951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.092050 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:07.092338 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.092836 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.095390 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.095832 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.095973 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.096066 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.096372 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.096462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:55:07.096669 28145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9ad3ca4da645d19bc47dd9e152424c" candidate_uuid: "e6528b323b0540a8b6ce27d2068abce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" is_pre_election: true
I20260928 23:55:07.097030 28145 raft_consensus.cc:2454] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6528b323b0540a8b6ce27d2068abce9 in term 0.
I20260928 23:55:07.097191 27872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9ad3ca4da645d19bc47dd9e152424c" candidate_uuid: "e6528b323b0540a8b6ce27d2068abce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56800e01ac454625836872974887c3e3" is_pre_election: true
I20260928 23:55:07.097496 27872 raft_consensus.cc:2454] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6528b323b0540a8b6ce27d2068abce9 in term 0.
I20260928 23:55:07.097572 27943 leader_election.cc:302] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aded145201948e8a1af2d4a2bc1e3d1, e6528b323b0540a8b6ce27d2068abce9; no voters:
I20260928 23:55:07.097915 28231 raft_consensus.cc:2781] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:07.098003 28231 raft_consensus.cc:482] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:07.098039 28231 raft_consensus.cc:3037] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:07.099126 28231 raft_consensus.cc:504] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.099556 28231 leader_election.cc:288] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [CANDIDATE]: Term 1 election: Requested vote from peers 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281), 56800e01ac454625836872974887c3e3 (127.20.88.1:40145)
I20260928 23:55:07.099933 28145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9ad3ca4da645d19bc47dd9e152424c" candidate_uuid: "e6528b323b0540a8b6ce27d2068abce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
I20260928 23:55:07.100062 27872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9ad3ca4da645d19bc47dd9e152424c" candidate_uuid: "e6528b323b0540a8b6ce27d2068abce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56800e01ac454625836872974887c3e3"
I20260928 23:55:07.100090 28145 raft_consensus.cc:3037] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:07.100203 27872 raft_consensus.cc:3037] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:07.101305 28145 raft_consensus.cc:2454] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6528b323b0540a8b6ce27d2068abce9 in term 1.
I20260928 23:55:07.101349 27872 raft_consensus.cc:2454] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6528b323b0540a8b6ce27d2068abce9 in term 1.
I20260928 23:55:07.101714 27943 leader_election.cc:302] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aded145201948e8a1af2d4a2bc1e3d1, 56800e01ac454625836872974887c3e3, e6528b323b0540a8b6ce27d2068abce9; no voters:
I20260928 23:55:07.101897 28231 raft_consensus.cc:2781] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:07.102180 28231 raft_consensus.cc:686] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 1 LEADER]: Becoming Leader. State: Replica: e6528b323b0540a8b6ce27d2068abce9, State: Running, Role: LEADER
I20260928 23:55:07.102556 28231 consensus_queue.cc:237] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } }
I20260928 23:55:07.106695 26954 catalog_manager.cc:5495] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "e6528b323b0540a8b6ce27d2068abce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } health_report { overall_health: HEALTHY } } }
I20260928 23:55:07.134490 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.130517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.20.88.2:42383)
0928 23:55:07.130791 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.130795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.130823 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:55:07.130933 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.130938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.131033 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.131146 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:07.131393 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.131935 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.132898 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.133143 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.133259 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.133594 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:07.134061 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:07.134203 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.134265 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.134330 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 23:55:07.141340 27872 raft_consensus.cc:1261] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Refusing update from remote peer e6528b323b0540a8b6ce27d2068abce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:07.141340 28145 raft_consensus.cc:1261] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Refusing update from remote peer e6528b323b0540a8b6ce27d2068abce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:07.142256 28231 consensus_queue.cc:1035] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:07.142488 28234 consensus_queue.cc:1035] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:07.152514 28239 mvcc.cc:205] Tried to move back new op lower bound from 7334460240441823232 to 7334460240295202816. Current Snapshot: MvccSnapshot[applied={T|T < 7334460240441823232}]
I20260928 23:55:07.164182 26953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:07.164399 26953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:07.169605 26953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0591079a0cfe4a50a667a39a1ea6c23d: Sending DeleteTablet for 3 replicas of tablet 0f9ad3ca4da645d19bc47dd9e152424c
I20260928 23:55:07.170743 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:07 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260928 23:55:07.170820 27989 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:07 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:34648
I20260928 23:55:07.171350 28247 tablet_replica.cc:330] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: stopping tablet replica
I20260928 23:55:07.171337 27852 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f9ad3ca4da645d19bc47dd9e152424c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:07 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:43954
I20260928 23:55:07.171687 28248 tablet_replica.cc:330] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: stopping tablet replica
I20260928 23:55:07.171806 28249 tablet_replica.cc:330] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: stopping tablet replica
I20260928 23:55:07.172207 28247 raft_consensus.cc:2229] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:07.172209 28249 raft_consensus.cc:2229] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:07.172559 28248 raft_consensus.cc:2229] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:07.172622 28249 raft_consensus.cc:2258] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:07.173029 28248 raft_consensus.cc:2258] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:07.173485 28247 raft_consensus.cc:2258] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:07.174661 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 26922
I20260928 23:55:07.175602 28247 ts_tablet_manager.cc:1905] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:07.176433 28248 ts_tablet_manager.cc:1905] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:07.176728 28249 ts_tablet_manager.cc:1905] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:07.178664 28247 ts_tablet_manager.cc:1918] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:07.178819 28247 log.cc:1196] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/0f9ad3ca4da645d19bc47dd9e152424c
I20260928 23:55:07.179172 28247 ts_tablet_manager.cc:1939] T 0f9ad3ca4da645d19bc47dd9e152424c P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting consensus metadata
I20260928 23:55:07.182413 28248 ts_tablet_manager.cc:1918] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 23:55:07.183579 28079 connection.cc:535] server connection from 127.0.0.1:46854 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:55:07.184643 28249 ts_tablet_manager.cc:1918] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:07.184859 28249 log.cc:1196] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/0f9ad3ca4da645d19bc47dd9e152424c
I20260928 23:55:07.185293 28249 ts_tablet_manager.cc:1939] T 0f9ad3ca4da645d19bc47dd9e152424c P 56800e01ac454625836872974887c3e3: Deleting consensus metadata
I20260928 23:55:07.185428 28248 log.cc:1196] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/0f9ad3ca4da645d19bc47dd9e152424c
W20260928 23:55:07.186667 27807 connection.cc:441] server connection from 127.0.0.1:43954 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43954 (request call id 1) could send its response
I20260928 23:55:07.187041 28248 ts_tablet_manager.cc:1939] T 0f9ad3ca4da645d19bc47dd9e152424c P e6528b323b0540a8b6ce27d2068abce9: Deleting consensus metadata
W20260928 23:55:07.187999 27944 connection.cc:441] server connection from 127.0.0.1:34648 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34648 (request call id 1) could send its response
I20260928 23:55:07.195399 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.172886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.20.88.61:40149)
0928 23:55:07.173004 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.173008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.184570 (+ 11562us) server_negotiation.cc:407] Connection header received
0928 23:55:07.184777 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.184780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.184838 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.184890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:07.185917 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.186771 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.188182 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.188604 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.188734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.191058 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:07.191062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:07.191065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:07.191092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:07.192992 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.193476 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.193479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.193481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.193572 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.194107 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.194110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.194112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.194320 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:07.194420 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.195075 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.195202 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:55:07.200011 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.189083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.20.88.61:40149)
0928 23:55:07.189212 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.189216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.189738 (+ 522us) server_negotiation.cc:407] Connection header received
0928 23:55:07.189901 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.189904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.189950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.190011 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:07.190941 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.191769 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.193013 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.193354 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.193425 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.195060 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:07.195064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:07.195067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:07.195091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:07.197666 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.198483 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.198486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.198488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.198573 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.198860 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.198863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.198865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.199094 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:07.199210 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.199717 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.199835 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:55:07.201884 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.189543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.20.88.60:36131)
0928 23:55:07.189681 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.189685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.192259 (+ 2574us) server_negotiation.cc:407] Connection header received
0928 23:55:07.192486 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.192493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.192551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.192606 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:07.193494 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.194274 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.195371 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.195682 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.195759 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.197699 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:07.197703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:07.197706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:07.197732 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:07.199445 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.199937 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.199943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.199947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.200039 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.200869 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.200875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.200878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.201100 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:07.201214 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.201603 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.201722 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:55:07.204638 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:07.173204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.20.88.60:36131)
0928 23:55:07.173398 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:07.173403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:07.195883 (+ 22480us) server_negotiation.cc:407] Connection header received
0928 23:55:07.196054 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:07.196057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:07.196101 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:07.196154 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:07.197127 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.197894 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.199147 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:07.199539 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:07.199610 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:07.201258 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:07.201262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:07.201266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:07.201289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:07.202951 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.203348 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.203351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.203353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.203441 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:07.203735 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:07.203738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:07.203739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:07.203911 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:07.204070 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:07.204334 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 23:55:07.204463 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 23:55:08.158037 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.157415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:32865)
0928 23:55:08.157787 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.157910 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
W20260928 23:55:08.158404 27919 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:41035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.158466 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.158055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:42641)
0928 23:55:08.158314 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.158381 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
W20260928 23:55:08.158739 28056 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:41035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.164008 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.163601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:36979)
0928 23:55:08.163828 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.163920 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
W20260928 23:55:08.164333 28192 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:41035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.670220 28260 raft_consensus.cc:482] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0591079a0cfe4a50a667a39a1ea6c23d)
I20260928 23:55:08.670439 28261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0591079a0cfe4a50a667a39a1ea6c23d)
I20260928 23:55:08.670547 28260 raft_consensus.cc:504] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } }
I20260928 23:55:08.670674 28261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } }
I20260928 23:55:08.671824 28260 leader_election.cc:288] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035), 707eaa96c0b24bd2be5f7310ca10c680 (127.20.88.61:40149)
I20260928 23:55:08.672129 28261 leader_election.cc:288] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035), f485b0c71c404ef99a60244c59736e14 (127.20.88.60:36131)
I20260928 23:55:08.672384 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.671895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40788)
0928 23:55:08.672190 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.672287 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":71,"threads_started":1}
I20260928 23:55:08.672384 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.671883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40772)
0928 23:55:08.672188 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.672287 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
W20260928 23:55:08.672561 27541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
W20260928 23:55:08.672632 27724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.673700 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.673469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40806)
0928 23:55:08.673563 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.673622 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:55:08.673802 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.673185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40792)
0928 23:55:08.673659 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.673704 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416}
W20260928 23:55:08.673961 27541 leader_election.cc:334] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035): Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
W20260928 23:55:08.673960 27724 leader_election.cc:334] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035): Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.676765 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.671947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:36131 (local address 127.0.0.1:39446)
0928 23:55:08.672369 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.672383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.672530 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.673436 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.673439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.673458 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.673550 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.673556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.674787 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.674793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.675587 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.675596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.676290 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.676293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.676394 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.676452 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.676546 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.676624 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":17,"thread_start_us":54,"threads_started":1}
I20260928 23:55:08.676765 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.672438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.20.88.60:36131)
0928 23:55:08.673213 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.673219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.673234 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:08.673275 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.673278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.673328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.673423 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.673901 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.674687 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.675717 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.676139 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.676279 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.676341 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.676561 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.676636 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":360,"server-negotiator.queue_time_us":703,"spinlock_wait_cycles":4992,"thread_start_us":62,"threads_started":1}
I20260928 23:55:08.677034 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.672784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.20.88.61:40149)
0928 23:55:08.673051 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.673060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.673072 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:55:08.673114 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.673117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.673162 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.673251 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.673656 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.674487 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.675723 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.676141 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.676363 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.676477 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.676650 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.676720 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
I20260928 23:55:08.677202 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.672216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:40149 (local address 127.0.0.1:43630)
0928 23:55:08.672519 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.672532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.672622 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.673282 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.673285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.673320 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.673480 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.673490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.674604 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.674606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.675585 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.675595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.676409 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.676414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.676511 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.676536 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.676634 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.676712 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"spinlock_wait_cycles":8448,"thread_start_us":57,"threads_started":1}
I20260928 23:55:08.677297 27749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "707eaa96c0b24bd2be5f7310ca10c680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f485b0c71c404ef99a60244c59736e14" is_pre_election: true
I20260928 23:55:08.677579 27749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 707eaa96c0b24bd2be5f7310ca10c680 in term 1.
I20260928 23:55:08.677981 27543 leader_election.cc:302] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 707eaa96c0b24bd2be5f7310ca10c680, f485b0c71c404ef99a60244c59736e14; no voters: 0591079a0cfe4a50a667a39a1ea6c23d
I20260928 23:55:08.678210 28261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 23:55:08.678287 28261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0591079a0cfe4a50a667a39a1ea6c23d)
I20260928 23:55:08.678359 28261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:55:08.678573 27566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f485b0c71c404ef99a60244c59736e14" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "707eaa96c0b24bd2be5f7310ca10c680" is_pre_election: true
I20260928 23:55:08.679598 28261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } }
I20260928 23:55:08.679939 27566 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f485b0c71c404ef99a60244c59736e14 in current term 2: Already voted for candidate 707eaa96c0b24bd2be5f7310ca10c680 in this term.
I20260928 23:55:08.680078 28261 leader_election.cc:288] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 election: Requested vote from peers 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035), f485b0c71c404ef99a60244c59736e14 (127.20.88.60:36131)
I20260928 23:55:08.680447 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.680192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40812)
0928 23:55:08.680315 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.680371 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:55:08.680578 27749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "707eaa96c0b24bd2be5f7310ca10c680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f485b0c71c404ef99a60244c59736e14"
I20260928 23:55:08.680699 27749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:55:08.680724 27724 leader_election.cc:302] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f485b0c71c404ef99a60244c59736e14; no voters: 0591079a0cfe4a50a667a39a1ea6c23d, 707eaa96c0b24bd2be5f7310ca10c680
I20260928 23:55:08.681823 27749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 707eaa96c0b24bd2be5f7310ca10c680 in term 2.
I20260928 23:55:08.681991 28260 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:55:08.682164 27543 leader_election.cc:302] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707eaa96c0b24bd2be5f7310ca10c680, f485b0c71c404ef99a60244c59736e14; no voters:
I20260928 23:55:08.682360 28261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:55:08.682598 28261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [term 2 LEADER]: Becoming Leader. State: Replica: 707eaa96c0b24bd2be5f7310ca10c680, State: Running, Role: LEADER
I20260928 23:55:08.682881 28261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } }
I20260928 23:55:08.683981 28270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707eaa96c0b24bd2be5f7310ca10c680. Latest consensus state: current_term: 2 leader_uuid: "707eaa96c0b24bd2be5f7310ca10c680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:08.684108 28270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:08.684363 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.684008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40826)
0928 23:55:08.684175 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.684280 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 23:55:08.684468 27541 leader_election.cc:334] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035): Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
I20260928 23:55:08.684643 28272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:08.686793 28272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:08.687206 28272 catalog_manager.cc:1259] Loaded cluster ID: 8d77d4c16fbe4e2e9d25b5f59a67af22
I20260928 23:55:08.687279 28272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:08.687834 28272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:08.688212 28272 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Loaded TSK: 0
I20260928 23:55:08.689016 28272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:55:08.739542 27555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:08.742923 27555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:08.746482 27749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [term 2 FOLLOWER]: Refusing update from remote peer 707eaa96c0b24bd2be5f7310ca10c680: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 23:55:08.746583 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.746100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40842)
0928 23:55:08.746340 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.746416 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 23:55:08.747562 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.747275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.0.0.1:40854)
0928 23:55:08.747419 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.747492 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 23:55:08.747465 28261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 23:55:08.747786 27541 consensus_peers.cc:487] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 -> Peer 0591079a0cfe4a50a667a39a1ea6c23d (127.20.88.62:41035): Couldn't send request to peer 0591079a0cfe4a50a667a39a1ea6c23d. Status: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:55:08.749442 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707eaa96c0b24bd2be5f7310ca10c680. Latest consensus state: current_term: 2 leader_uuid: "707eaa96c0b24bd2be5f7310ca10c680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:08.749614 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:08.751150 28270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "707eaa96c0b24bd2be5f7310ca10c680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:08.751288 28270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:08.751572 28274 mvcc.cc:205] Tried to move back new op lower bound from 7334460247020052480 to 7334460246767316992. Current Snapshot: MvccSnapshot[applied={T|T < 7334460247020052480}]
I20260928 23:55:08.751835 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "707eaa96c0b24bd2be5f7310ca10c680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0591079a0cfe4a50a667a39a1ea6c23d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41035 } } peers { permanent_uuid: "707eaa96c0b24bd2be5f7310ca10c680" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 40149 } attrs { promote: false } } peers { permanent_uuid: "f485b0c71c404ef99a60244c59736e14" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 36131 } attrs { promote: false } } }
I20260928 23:55:08.751979 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f485b0c71c404ef99a60244c59736e14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:55:08.763736 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.760137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.20.88.1:40145)
0928 23:55:08.760435 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.760439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.760453 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:08.760503 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.760505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.760553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.760638 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.760924 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761663 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.762686 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763163 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.763341 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.763513 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.763592 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260928 23:55:08.763737 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.759783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:40145 (local address 127.0.0.1:57406)
0928 23:55:08.760053 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.760069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.760153 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.760669 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.760672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.760690 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.760777 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.760782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761771 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.761777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.762552 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.762561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763283 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.763373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.763397 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.763484 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.763561 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
I20260928 23:55:08.764473 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.759073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:42383 (local address 127.0.0.1:48316)
0928 23:55:08.759430 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.759446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.759563 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.759957 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.759971 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.759992 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.760116 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.760122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761416 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.761420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.762315 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.762324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763984 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.764143 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.764167 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.764257 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.764323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":52,"spinlock_wait_cycles":9856}
I20260928 23:55:08.764472 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.759382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43281 (local address 127.0.0.1:59826)
0928 23:55:08.759465 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.759478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.759563 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.760160 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.760163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.760181 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.760320 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.760327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761421 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.761425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.763029 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.763035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763967 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.764108 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.764134 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.764187 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.764316 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":850,"spinlock_wait_cycles":2816}
I20260928 23:55:08.764516 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.759186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.20.88.2:42383)
0928 23:55:08.759678 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.759685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.759702 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:55:08.759767 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.759771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.759825 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.760278 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.760395 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761144 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763173 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763829 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.764076 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.764139 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.764279 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.764361 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260928 23:55:08.764955 27852 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 (DEFAULT_TABLE table=test-table [id=10d55c7b2f5f4c8d8b0188b04f765419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:08.765115 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.759631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.20.88.3:43281)
0928 23:55:08.759882 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.759886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.759919 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:55:08.759977 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.759980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.760029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.760428 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.760551 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.761294 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.763149 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.763823 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.764807 (+ 984us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.764872 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.764908 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.764981 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
I20260928 23:55:08.765298 27852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:08.765254 27989 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 (DEFAULT_TABLE table=test-table [id=10d55c7b2f5f4c8d8b0188b04f765419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:08.765523 27989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:08.765690 28125 tablet_service.cc:1467] Processing CreateTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 (DEFAULT_TABLE table=test-table [id=10d55c7b2f5f4c8d8b0188b04f765419]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:08.765928 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:08.768087 28284 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: Bootstrap starting.
I20260928 23:55:08.768672 28285 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Bootstrap starting.
I20260928 23:55:08.768955 28283 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Bootstrap starting.
I20260928 23:55:08.769135 28284 tablet_bootstrap.cc:654] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:08.769671 28285 tablet_bootstrap.cc:654] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:08.770057 28283 tablet_bootstrap.cc:654] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:08.770494 28284 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: No bootstrap required, opened a new log
I20260928 23:55:08.770592 28284 ts_tablet_manager.cc:1397] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:08.770949 28285 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: No bootstrap required, opened a new log
I20260928 23:55:08.771044 28285 ts_tablet_manager.cc:1397] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:55:08.771178 28284 raft_consensus.cc:348] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.771286 28284 raft_consensus.cc:374] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:08.771283 28283 tablet_bootstrap.cc:492] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: No bootstrap required, opened a new log
I20260928 23:55:08.771308 28284 raft_consensus.cc:729] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56800e01ac454625836872974887c3e3, State: Initialized, Role: FOLLOWER
I20260928 23:55:08.771360 28283 ts_tablet_manager.cc:1397] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:55:08.771448 28284 consensus_queue.cc:260] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.771495 28285 raft_consensus.cc:348] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.771598 28285 raft_consensus.cc:374] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:08.771629 28285 raft_consensus.cc:729] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aded145201948e8a1af2d4a2bc1e3d1, State: Initialized, Role: FOLLOWER
I20260928 23:55:08.771783 28284 ts_tablet_manager.cc:1428] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 23:55:08.771782 28285 consensus_queue.cc:260] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.771792 28283 raft_consensus.cc:348] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.771893 28283 raft_consensus.cc:374] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:08.771927 28283 raft_consensus.cc:729] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6528b323b0540a8b6ce27d2068abce9, State: Initialized, Role: FOLLOWER
I20260928 23:55:08.772171 28285 ts_tablet_manager.cc:1428] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:08.772104 28283 consensus_queue.cc:260] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.772481 28283 ts_tablet_manager.cc:1428] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:08.774870 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.773153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:59255)
0928 23:55:08.774619 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.774756 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":89,"threads_started":1}
I20260928 23:55:08.775584 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.774909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:33085)
0928 23:55:08.775397 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.775468 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":100,"threads_started":1}
I20260928 23:55:08.775898 27920 heartbeater.cc:503] Master 127.20.88.61:40149 was elected leader, sending a full tablet report...
I20260928 23:55:08.776823 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.772288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:32779)
0928 23:55:08.776639 (+ 4351us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.776701 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4284,"thread_start_us":94,"threads_started":1}
I20260928 23:55:08.776919 28057 heartbeater.cc:503] Master 127.20.88.61:40149 was elected leader, sending a full tablet report...
I20260928 23:55:08.777691 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.777296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:59243)
0928 23:55:08.777574 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.777626 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260928 23:55:08.778416 28193 heartbeater.cc:503] Master 127.20.88.61:40149 was elected leader, sending a full tablet report...
I20260928 23:55:08.778937 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.776369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:41913)
0928 23:55:08.776757 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.777279 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260928 23:55:08.779413 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.777863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:37279)
0928 23:55:08.779223 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.779306 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":256,"threads_started":1}
I20260928 23:55:08.783876 28289 raft_consensus.cc:482] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:08.784058 28289 raft_consensus.cc:504] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.785501 28289 leader_election.cc:288] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383), 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
W20260928 23:55:08.789645 27921 tablet.cc:2369] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:55:08.790548 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.785817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:51585 (local address 127.20.88.2:42383)
0928 23:55:08.785978 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.785983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.786001 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:55:08.786058 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.786061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.786119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.786222 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.786651 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.787476 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.789241 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.789846 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.789959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.790035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.790283 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.790374 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 23:55:08.790568 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.786070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57493 (local address 127.20.88.3:43281)
0928 23:55:08.786213 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.786218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.786426 (+ 208us) server_negotiation.cc:407] Connection header received
0928 23:55:08.786541 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.786543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.786608 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.786689 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:08.786979 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.787678 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.789241 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.789865 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.789959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.790036 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.790342 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.790434 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:55:08.790628 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.785956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:43281 (local address 127.20.88.1:57493)
0928 23:55:08.786324 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.786339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.786436 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.786740 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.786743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.786763 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.786862 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.786866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.787778 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.787782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.789106 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.789118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.790107 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.790110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.790191 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.790209 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.790308 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.790381 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":75,"thread_start_us":164,"threads_started":1}
I20260928 23:55:08.791182 28145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc64482a0a4d2f804d8ab6a2ed2ba9" candidate_uuid: "56800e01ac454625836872974887c3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" is_pre_election: true
I20260928 23:55:08.791348 28145 raft_consensus.cc:2454] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56800e01ac454625836872974887c3e3 in term 0.
I20260928 23:55:08.791755 27806 leader_election.cc:302] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aded145201948e8a1af2d4a2bc1e3d1, 56800e01ac454625836872974887c3e3; no voters:
I20260928 23:55:08.792060 28289 raft_consensus.cc:2781] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:08.792096 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.785713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:42383 (local address 127.20.88.1:51585)
0928 23:55:08.785827 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.785844 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:08.785938 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:08.786358 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:08.786362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:08.786386 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:08.786526 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.786532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.787594 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.787597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.789108 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:08.789118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.790019 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.790023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:08.790136 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.790162 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:55:08.790247 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:55:08.790374 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":40}
I20260928 23:55:08.792140 28289 raft_consensus.cc:482] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:08.792191 28289 raft_consensus.cc:3037] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:08.792687 28009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc64482a0a4d2f804d8ab6a2ed2ba9" candidate_uuid: "56800e01ac454625836872974887c3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6528b323b0540a8b6ce27d2068abce9" is_pre_election: true
I20260928 23:55:08.792963 28009 raft_consensus.cc:2454] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56800e01ac454625836872974887c3e3 in term 0.
I20260928 23:55:08.793622 28289 raft_consensus.cc:504] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.794003 28289 leader_election.cc:288] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [CANDIDATE]: Term 1 election: Requested vote from peers e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383), 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281)
I20260928 23:55:08.794363 28009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc64482a0a4d2f804d8ab6a2ed2ba9" candidate_uuid: "56800e01ac454625836872974887c3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6528b323b0540a8b6ce27d2068abce9"
I20260928 23:55:08.794477 28145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77dc64482a0a4d2f804d8ab6a2ed2ba9" candidate_uuid: "56800e01ac454625836872974887c3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aded145201948e8a1af2d4a2bc1e3d1"
I20260928 23:55:08.794508 28009 raft_consensus.cc:3037] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:08.794615 28145 raft_consensus.cc:3037] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:08.795542 28145 raft_consensus.cc:2454] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56800e01ac454625836872974887c3e3 in term 1.
I20260928 23:55:08.795552 28009 raft_consensus.cc:2454] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56800e01ac454625836872974887c3e3 in term 1.
I20260928 23:55:08.795849 27806 leader_election.cc:302] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56800e01ac454625836872974887c3e3, e6528b323b0540a8b6ce27d2068abce9; no voters:
I20260928 23:55:08.796032 28289 raft_consensus.cc:2781] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:08.796316 28289 raft_consensus.cc:686] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 1 LEADER]: Becoming Leader. State: Replica: 56800e01ac454625836872974887c3e3, State: Running, Role: LEADER
I20260928 23:55:08.796613 28289 consensus_queue.cc:237] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } }
I20260928 23:55:08.797255 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.796894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:33277)
0928 23:55:08.797071 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.797157 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 23:55:08.799437 27555 catalog_manager.cc:5495] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56800e01ac454625836872974887c3e3 (127.20.88.1). New cstate: current_term: 1 leader_uuid: "56800e01ac454625836872974887c3e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56800e01ac454625836872974887c3e3" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 40145 } health_report { overall_health: HEALTHY } } }
I20260928 23:55:08.820703 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.817073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.20.88.1:40145)
0928 23:55:08.817206 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:08.817209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:08.817224 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:08.817324 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:08.817326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:08.817394 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:08.817497 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:08.817721 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.818262 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.819155 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:08.819388 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:08.819479 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:08.819703 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:08.820194 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:08.820369 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:08.820519 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:55:08.820577 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:55:08.823702 28145 raft_consensus.cc:1261] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Refusing update from remote peer 56800e01ac454625836872974887c3e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:08.823961 28009 raft_consensus.cc:1261] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Refusing update from remote peer 56800e01ac454625836872974887c3e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:08.824265 28289 consensus_queue.cc:1035] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aded145201948e8a1af2d4a2bc1e3d1" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:08.824596 28297 consensus_queue.cc:1035] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6528b323b0540a8b6ce27d2068abce9" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:08.826217 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.825899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:39007)
0928 23:55:08.826025 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.826094 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:55:08.827625 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.827341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:52765)
0928 23:55:08.827476 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.827542 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:55:08.829378 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.829098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.3:36295)
0928 23:55:08.829246 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.829306 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 23:55:08.830756 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.828670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:37991)
0928 23:55:08.830615 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.830682 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1856}
I20260928 23:55:08.835295 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.835017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.2:53271)
0928 23:55:08.835164 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.835225 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 23:55:08.836711 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.834510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41035 (local address 127.20.88.1:48277)
0928 23:55:08.836558 (+ 2048us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.836624 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:41035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1978}
I20260928 23:55:08.837147 28305 mvcc.cc:205] Tried to move back new op lower bound from 7334460247336480768 to 7334460247233544192. Current Snapshot: MvccSnapshot[applied={T|T < 7334460247336480768}]
I20260928 23:55:08.838507 27555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:08.838670 27555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:08.842919 27555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 707eaa96c0b24bd2be5f7310ca10c680: Sending DeleteTablet for 3 replicas of tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9
I20260928 23:55:08.843550 27989 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:08 UTC) from {username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:48316
I20260928 23:55:08.843920 28309 tablet_replica.cc:330] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: stopping tablet replica
I20260928 23:55:08.844075 28309 raft_consensus.cc:2229] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:08.844205 28309 raft_consensus.cc:2258] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:08.844308 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:08 UTC) from {username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:59826
I20260928 23:55:08.845058 28309 ts_tablet_manager.cc:1905] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:08.845063 27852 tablet_service.cc:1514] Processing DeleteTablet for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:08 UTC) from {username='oryx', principal='oryx/127.20.88.61@KRBTEST.COM'} at 127.0.0.1:57406
I20260928 23:55:08.845482 28311 tablet_replica.cc:330] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3: stopping tablet replica
I20260928 23:55:08.845602 28311 raft_consensus.cc:2229] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:08.845937 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27700
I20260928 23:55:08.845964 28311 raft_consensus.cc:2258] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 56800e01ac454625836872974887c3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:08.849813 28310 tablet_replica.cc:330] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: stopping tablet replica
I20260928 23:55:08.849865 28309 ts_tablet_manager.cc:1918] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:08.849936 28309 log.cc:1196] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/77dc64482a0a4d2f804d8ab6a2ed2ba9
I20260928 23:55:08.849979 28310 raft_consensus.cc:2229] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:08.850139 28310 raft_consensus.cc:2258] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:08.850293 28309 ts_tablet_manager.cc:1939] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P e6528b323b0540a8b6ce27d2068abce9: Deleting consensus metadata
I20260928 23:55:08.850909 28310 ts_tablet_manager.cc:1905] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:08.851135 27541 catalog_manager.cc:4841] TS e6528b323b0540a8b6ce27d2068abce9 (127.20.88.2:42383): tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 (table test-table [id=10d55c7b2f5f4c8d8b0188b04f765419]) successfully deleted
I20260928 23:55:08.853003 28310 ts_tablet_manager.cc:1918] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:08.853070 28310 log.cc:1196] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/77dc64482a0a4d2f804d8ab6a2ed2ba9
I20260928 23:55:08.853360 28310 ts_tablet_manager.cc:1939] T 77dc64482a0a4d2f804d8ab6a2ed2ba9 P 2aded145201948e8a1af2d4a2bc1e3d1: Deleting consensus metadata
I20260928 23:55:08.854043 27541 catalog_manager.cc:4841] TS 2aded145201948e8a1af2d4a2bc1e3d1 (127.20.88.3:43281): tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9 (table test-table [id=10d55c7b2f5f4c8d8b0188b04f765419]) successfully deleted
I20260928 23:55:08.855372 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27926
I20260928 23:55:08.855696 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:08.855397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:40145 (local address 127.0.0.1:57424)
0928 23:55:08.855537 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:08.855595 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:55:08.856454 27540 catalog_manager.cc:4584] TS 56800e01ac454625836872974887c3e3 (127.20.88.1:40145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77dc64482a0a4d2f804d8ab6a2ed2ba9: Network error: Client connection negotiation failed: client connection to 127.20.88.1:40145: connect: Connection refused (error 111)
I20260928 23:55:08.862617 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 28062
I20260928 23:55:08.868584 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27418
I20260928 23:55:08.874634 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 27520
2026-09-28T23:55:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5498 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:08 dist-test-slave-wvpw krb5kdc[28314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:08 dist-test-slave-wvpw krb5kdc[28314](info): set up 2 sockets
Sep 28 23:55:08 dist-test-slave-wvpw krb5kdc[28314](info): commencing operation
krb5kdc: starting...
W20260928 23:55:09.925559 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab.
Entry for principal oryx/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab.
I20260928 23:55:10.056564 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32837
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab
--principal=oryx/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:10.179868 28327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:10.180179 28327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:10.180223 28327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:10.184497 28327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:10.184561 28327 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:55:10.184578 28327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:10.184597 28327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:10.184662 28327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:10.184692 28327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:10.190303 28327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:32837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.62.keytab
--principal=oryx/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:10.191490 28327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:10.192656 28327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:10.201604 28333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.201604 28335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.201642 28332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.202085 28327 server_base.cc:1034] running on GCE node
I20260928 23:55:10.202749 28327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:10.203938 28327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:10.205132 28327 hybrid_clock.cc:648] HybridClock initialized: now 1790639710205094 us; error 68 us; skew 500 ppm
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:10.207952 28327 init.cc:377] Logged in from keytab as oryx/127.20.88.62@KRBTEST.COM (short username oryx)
I20260928 23:55:10.208914 28327 webserver.cc:460] Webserver started at http://127.20.88.62:45033/ using document root <none> and password file <none>
I20260928 23:55:10.209530 28327 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:10.209628 28327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:10.209872 28327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:10.212028 28327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "787c7906e9d14a0b8c8248de978be6a2"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "2d8f318acfa8c81f40badb6cbc6fd749"
server_key_iv: "b7670426727cec0043e4aa94adc4e1a0"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.212716 28327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "787c7906e9d14a0b8c8248de978be6a2"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "2d8f318acfa8c81f40badb6cbc6fd749"
server_key_iv: "b7670426727cec0043e4aa94adc4e1a0"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.216907 28327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:10.219777 28342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:10.221477 28327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:10.221685 28327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "787c7906e9d14a0b8c8248de978be6a2"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "2d8f318acfa8c81f40badb6cbc6fd749"
server_key_iv: "b7670426727cec0043e4aa94adc4e1a0"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.221835 28327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:10.242043 28327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:10.245767 28327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:10.246006 28327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:10.254907 28327 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:32837
I20260928 23:55:10.254912 28404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:32837 every 8 connection(s)
I20260928 23:55:10.256685 28327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:10.260015 28405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:10.264436 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 28327
I20260928 23:55:10.264618 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:10.264933 20832 external_mini_cluster.cc:1428] Setting key 07a51ba0e582e2356a90f1469645fd63
I20260928 23:55:10.266673 28405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Bootstrap starting.
I20260928 23:55:10.269116 28405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:10.269919 28405 log.cc:824] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:10.271953 28405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: No bootstrap required, opened a new log
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:10.275839 28405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } }
I20260928 23:55:10.276108 28405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:10.276182 28405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787c7906e9d14a0b8c8248de978be6a2, State: Initialized, Role: FOLLOWER
I20260928 23:55:10.276698 28405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } }
I20260928 23:55:10.276865 28405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:10.276953 28405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:10.277092 28405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:10.278110 28405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } }
I20260928 23:55:10.278487 28405 leader_election.cc:302] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787c7906e9d14a0b8c8248de978be6a2; no voters:
I20260928 23:55:10.278861 28405 leader_election.cc:288] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:10.278942 28410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:10.279145 28410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [term 1 LEADER]: Becoming Leader. State: Replica: 787c7906e9d14a0b8c8248de978be6a2, State: Running, Role: LEADER
I20260928 23:55:10.279573 28410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } }
I20260928 23:55:10.280373 28405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:10.281733 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.266532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.20.88.62:32837)
0928 23:55:10.267082 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:10.267099 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:10.267139 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:55:10.267896 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:10.267916 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:10.268387 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:10.268772 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:10.269697 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.270885 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.271842 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.272506 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.272702 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.275671 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:10.275677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:10.275692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:10.275723 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:10.277918 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.278503 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.278510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.278515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.278675 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.279036 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.279040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.279044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.279740 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:10.279918 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:10.280377 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 23:55:10.280805 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":153,"threads_started":1}
I20260928 23:55:10.282785 28411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787c7906e9d14a0b8c8248de978be6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } } }
I20260928 23:55:10.283933 28411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:10.282555 28412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787c7906e9d14a0b8c8248de978be6a2. Latest consensus state: current_term: 1 leader_uuid: "787c7906e9d14a0b8c8248de978be6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787c7906e9d14a0b8c8248de978be6a2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 32837 } } }
I20260928 23:55:10.284183 28412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2 [sys.catalog]: This master's current role is: LEADER
W20260928 23:55:10.285519 28424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:55:10.285586 28424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:55:10.285708 28427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:10.286440 28427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:10.292649 28427 catalog_manager.cc:1347] Generated new cluster ID: f346afbf183740e09a7b6b107c8cb705
I20260928 23:55:10.292737 28427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:10.306541 28427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:10.307994 28427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:10.322547 28427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Generated new TSK 0
I20260928 23:55:10.323585 28427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
I20260928 23:55:10.355960 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32837
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:10.483381 28431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:10.483709 28431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:10.483794 28431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:10.489327 28431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:10.489420 28431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:10.489498 28431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:55:10.489599 28431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:10.495409 28431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:10.496642 28431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:10.497774 28431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:10.507824 28439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.507979 28431 server_base.cc:1034] running on GCE node
W20260928 23:55:10.507826 28437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.507826 28436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.508628 28431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:10.509341 28431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:10.510519 28431 hybrid_clock.cc:648] HybridClock initialized: now 1790639710510481 us; error 49 us; skew 500 ppm
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:10.513586 28431 init.cc:377] Logged in from keytab as oryx/127.20.88.1@KRBTEST.COM (short username oryx)
I20260928 23:55:10.514609 28431 webserver.cc:460] Webserver started at http://127.20.88.1:35645/ using document root <none> and password file <none>
I20260928 23:55:10.515210 28431 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:10.515275 28431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:10.515508 28431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:10.517755 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "9104d649006240dc8899aaed93ed4cef"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "868b7b0e42f31d7f15437014a6518f38"
server_key_iv: "86b55c701f254cdae28ff0312e3680cb"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.518347 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "9104d649006240dc8899aaed93ed4cef"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "868b7b0e42f31d7f15437014a6518f38"
server_key_iv: "86b55c701f254cdae28ff0312e3680cb"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.522428 28431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:55:10.525159 28446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:10.526654 28431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:55:10.526809 28431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "9104d649006240dc8899aaed93ed4cef"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "868b7b0e42f31d7f15437014a6518f38"
server_key_iv: "86b55c701f254cdae28ff0312e3680cb"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.526969 28431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:10.556511 28431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:10.559780 28431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:10.559957 28431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:10.561444 28431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:10.561506 28431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:10.561599 28431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:10.561637 28431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:10.572888 28431 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35631
I20260928 23:55:10.572954 28561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35631 every 8 connection(s)
I20260928 23:55:10.573859 28431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:10.574309 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 28431
I20260928 23:55:10.574450 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:10.574783 20832 external_mini_cluster.cc:1428] Setting key aca1512468d937553f695a3e8c7ba512
I20260928 23:55:10.577157 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.562532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:35685 (local address 127.20.88.62:32837)
0928 23:55:10.562695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:10.562700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:10.563634 (+ 934us) server_negotiation.cc:407] Connection header received
0928 23:55:10.564907 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:10.564911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:10.564984 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:10.565101 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:10.566702 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.567268 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.568812 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.569075 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.569206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.572780 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:10.572784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:10.572787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:10.572822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:10.574719 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.575384 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.575391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.575393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.575500 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.575860 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.575863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.575865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.576097 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:10.576201 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:10.576878 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 23:55:10.577003 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:55:10.578493 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.562971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.1:35685)
0928 23:55:10.563445 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:10.563498 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:10.564643 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:10.565363 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:10.565376 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:10.565776 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:10.566522 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.566538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.567421 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.567424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.568686 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.568701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.569270 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.569275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.569471 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.570801 (+ 1330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:10.570818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:10.572574 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:10.574878 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.574884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.574899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.575247 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.575602 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.575604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.575606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.575749 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.576232 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:10.576240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:10.576597 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 23:55:10.576812 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 23:55:10.577238 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260928 23:55:10.580390 28562 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32837
I20260928 23:55:10.580602 28562 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:10.581424 28562 heartbeater.cc:511] Master 127.20.88.62:32837 requested a full tablet report, sending...
I20260928 23:55:10.583433 28358 ts_manager.cc:194] Registered new tserver with Master: 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631)
I20260928 23:55:10.584929 28358 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:35685
WARNING: no policy specified for oryx/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.2@KRBTEST.COM" created.
I20260928 23:55:10.592519 28569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.583108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.1:38589)
0928 23:55:10.583523 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:10.583542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:10.583666 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:10.584094 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:10.584097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:10.584374 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:10.584533 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.584543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.585832 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.585838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.586793 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.586804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.587483 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.587490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.587596 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.588166 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:10.588178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:10.588633 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:10.590779 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.590785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.590787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.591087 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.591373 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.591379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.591384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.591490 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.591873 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:10.591879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:10.592034 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 23:55:10.592166 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 23:55:10.592358 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
I20260928 23:55:10.592729 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.583225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38589 (local address 127.20.88.62:32837)
0928 23:55:10.583384 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:10.583390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:10.583639 (+ 249us) server_negotiation.cc:407] Connection header received
0928 23:55:10.583775 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:10.583781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:10.583835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:10.583943 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:10.584689 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.585532 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.586934 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.587265 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.587439 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.588865 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:10.588873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:10.588877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:10.588931 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:10.590645 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.591197 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.591200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.591201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.591277 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.591583 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.591586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.591587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.591773 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:10.591847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:10.592329 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 23:55:10.592493 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
I20260928 23:55:10.608342 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32837
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:10.727095 28571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:10.727404 28571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:10.727451 28571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:10.731787 28571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:10.731868 28571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:10.731912 28571 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:55:10.731946 28571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:10.737648 28571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:10.738744 28571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:10.739748 28571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:10.750371 28579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.750362 28577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.750388 28576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.750653 28571 server_base.cc:1034] running on GCE node
I20260928 23:55:10.751333 28571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:10.752094 28571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:10.753290 28571 hybrid_clock.cc:648] HybridClock initialized: now 1790639710753274 us; error 40 us; skew 500 ppm
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:10.756402 28571 init.cc:377] Logged in from keytab as oryx/127.20.88.2@KRBTEST.COM (short username oryx)
I20260928 23:55:10.757494 28571 webserver.cc:460] Webserver started at http://127.20.88.2:40227/ using document root <none> and password file <none>
I20260928 23:55:10.758119 28571 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:10.758219 28571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:10.758469 28571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:10.760735 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "440b49a976fb4dab92bae5952b10404f"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "76094cd18618718e157e978c91b84ed6"
server_key_iv: "859c71615888b2d862c7d7a9f907152f"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.761349 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "440b49a976fb4dab92bae5952b10404f"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "76094cd18618718e157e978c91b84ed6"
server_key_iv: "859c71615888b2d862c7d7a9f907152f"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.765458 28571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:10.768179 28586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:10.769696 28571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:10.769878 28571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "440b49a976fb4dab92bae5952b10404f"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "76094cd18618718e157e978c91b84ed6"
server_key_iv: "859c71615888b2d862c7d7a9f907152f"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.770023 28571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:10.790648 28571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:10.794114 28571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:10.794353 28571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:10.795960 28571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:10.796047 28571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:10.796128 28571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:10.796171 28571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:10.807476 28571 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:46135
I20260928 23:55:10.807511 28701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:46135 every 8 connection(s)
I20260928 23:55:10.808507 28571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:10.810978 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.796897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:34775 (local address 127.20.88.62:32837)
0928 23:55:10.797097 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:10.797104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:10.798027 (+ 923us) server_negotiation.cc:407] Connection header received
0928 23:55:10.799223 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:10.799227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:10.799289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:10.799379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:10.800875 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.801416 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.803331 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.803769 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.803904 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.806992 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:10.806997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:10.807000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:10.807034 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:10.808806 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.809398 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.809402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.809403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.809498 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.809859 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.809864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.809865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.810074 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:10.810180 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:10.810672 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 23:55:10.810805 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:55:10.811775 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.797343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.2:34775)
0928 23:55:10.797805 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:10.797859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:10.798942 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:10.799632 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:10.799641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:10.800025 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:10.800726 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.800749 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.801555 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.801559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.803159 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.803170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.803989 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.803995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.804220 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.804979 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:10.804998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:10.806778 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:10.808939 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.808945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.808961 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.809304 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.809601 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.809604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.809606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.809766 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.810180 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:10.810187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:10.810435 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 23:55:10.810619 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:55:10.810951 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260928 23:55:10.813014 28702 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32837
I20260928 23:55:10.813187 28702 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:10.813768 28702 heartbeater.cc:511] Master 127.20.88.62:32837 requested a full tablet report, sending...
I20260928 23:55:10.814881 28359 ts_manager.cc:194] Registered new tserver with Master: 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135)
I20260928 23:55:10.815759 28359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:34775
I20260928 23:55:10.815915 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 28571
I20260928 23:55:10.816030 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:10.816375 20832 external_mini_cluster.cc:1428] Setting key 5c2366fbac325ba43f54bda6bb9264fc
I20260928 23:55:10.823189 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.815067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:58823 (local address 127.20.88.62:32837)
0928 23:55:10.815193 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:10.815197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:10.815438 (+ 241us) server_negotiation.cc:407] Connection header received
0928 23:55:10.815601 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:10.815603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:10.815664 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:10.815748 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:10.816304 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.817055 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.818092 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.818322 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.818430 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.819763 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:10.819766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:10.819768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:10.819791 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:10.821290 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.821816 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.821819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.821820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.821891 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:10.822217 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:10.822220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:10.822221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:10.822467 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:10.822575 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:10.822956 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 23:55:10.823071 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:55:10.823339 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:10.814949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.2:58823)
0928 23:55:10.815348 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:10.815363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:10.815464 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:10.815784 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:10.815788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:10.816004 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:10.816139 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.816147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.817197 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.817200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.817970 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:10.817980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:10.818433 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:10.818436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:10.818536 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:10.819097 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:10.819108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:10.819619 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:10.821404 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.821411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.821416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.821701 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.821981 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:10.821984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:10.821988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:10.822093 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:10.822602 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:10.822609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:10.822737 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 23:55:10.822870 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 23:55:10.823055 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
I20260928 23:55:10.847088 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:32837
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:10.964349 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:10.964672 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:10.964768 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:10.969018 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:10.969106 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:10.969151 28711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:55:10.969252 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:10.975044 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.88.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:10.976365 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:10.977530 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:10.988656 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.988699 28711 server_base.cc:1034] running on GCE node
W20260928 23:55:10.988620 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:10.988624 28717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:10.989352 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:10.990094 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:10.991313 28711 hybrid_clock.cc:648] HybridClock initialized: now 1790639710991297 us; error 30 us; skew 500 ppm
Sep 28 23:55:10 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:10.994517 28711 init.cc:377] Logged in from keytab as oryx/127.20.88.3@KRBTEST.COM (short username oryx)
I20260928 23:55:10.995597 28711 webserver.cc:460] Webserver started at http://127.20.88.3:40885/ using document root <none> and password file <none>
I20260928 23:55:10.996315 28711 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:10.996413 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:10.996657 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:10.999123 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "7781005b6ab34b659e5e2c0682607344"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "c03e27d91da264e53bd44aa7653d9bed"
server_key_iv: "0b47b9f3365969df11af47e3407c9796"
server_key_version: "encryptionkey@0"
I20260928 23:55:10.999797 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "7781005b6ab34b659e5e2c0682607344"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "c03e27d91da264e53bd44aa7653d9bed"
server_key_iv: "0b47b9f3365969df11af47e3407c9796"
server_key_version: "encryptionkey@0"
I20260928 23:55:11.004608 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 23:55:11.007908 28726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:11.009580 28711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:11.009804 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "7781005b6ab34b659e5e2c0682607344"
format_stamp: "Formatted at 2026-09-28 23:55:10 on dist-test-slave-wvpw"
server_key: "c03e27d91da264e53bd44aa7653d9bed"
server_key_iv: "0b47b9f3365969df11af47e3407c9796"
server_key_version: "encryptionkey@0"
I20260928 23:55:11.009960 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:11.022058 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:11.025449 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:11.025684 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:11.027300 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:11.027379 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:11.027449 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:11.027498 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:11.038903 28711 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:41891
I20260928 23:55:11.038998 28841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:41891 every 8 connection(s)
I20260928 23:55:11.040114 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:11.042054 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.028275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:56737 (local address 127.20.88.62:32837)
0928 23:55:11.028439 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.028446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.029429 (+ 983us) server_negotiation.cc:407] Connection header received
0928 23:55:11.030965 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.030969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.031062 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.031189 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.032559 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.033148 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.034417 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.034651 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.034776 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.037954 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:11.037957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:11.037959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:11.037985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:11.039806 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.040414 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.040418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.040420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.040523 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.040925 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.040929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.040930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.041170 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:11.041281 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.041822 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.041933 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:55:11.042893 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.028729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.3:56737)
0928 23:55:11.029227 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.029291 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.030619 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.031373 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.031385 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.031800 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:11.032388 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.032432 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.033272 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.033275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.034274 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.034283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.034829 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.034833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.035020 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.035703 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:11.035720 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:11.037749 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:11.039945 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.039950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.039962 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.040286 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.040630 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.040633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.040635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.040803 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.041290 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:11.041297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:11.041601 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.041755 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.042013 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20260928 23:55:11.044104 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 28711
I20260928 23:55:11.044266 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:11.044282 28842 heartbeater.cc:347] Connected to a master server at 127.20.88.62:32837
I20260928 23:55:11.044425 28842 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:11.044680 20832 external_mini_cluster.cc:1428] Setting key ea140df337884ecf11fe608d4f17b1c7
I20260928 23:55:11.045050 28842 heartbeater.cc:511] Master 127.20.88.62:32837 requested a full tablet report, sending...
I20260928 23:55:11.046288 28358 ts_manager.cc:194] Registered new tserver with Master: 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.047088 28358 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:56737
I20260928 23:55:11.047253 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:55:11.055418 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.046863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53349 (local address 127.20.88.62:32837)
0928 23:55:11.046999 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.047002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.048456 (+ 1454us) server_negotiation.cc:407] Connection header received
0928 23:55:11.048622 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.048625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.048672 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.048759 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.049224 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.049733 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.050555 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.050776 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.050886 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.052060 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:11.052063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:11.052066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:11.052097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:11.053633 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.054169 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.054173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.054175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.054262 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.054548 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.054553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.054555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.054747 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:11.054854 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.055124 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.055238 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:55:11.055442 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.046796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.20.88.3:53349)
0928 23:55:11.048333 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.048357 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.048515 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.048775 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.048778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.049006 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:11.049107 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.049112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.049858 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.049861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.050433 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.050441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.050890 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.050894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.050993 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.051564 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:11.051576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:11.051919 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:11.053799 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.053803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.053806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.054064 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.054355 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.054359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.054361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.054454 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.054842 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:11.054845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:11.054949 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.055057 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.055222 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":87,"threads_started":1}
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:55:11.073431 28408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:11.066737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.20.88.62:32837)
0928 23:55:11.066901 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.066905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.066920 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:11.066965 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.066967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.067007 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.067086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.067842 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.068470 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.069399 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.069620 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.069713 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.073303 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37130: BlockingRecv error: recv got EOF from 127.0.0.1:37130 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:11.084830 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.074895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.20.88.62:32837)
0928 23:55:11.075128 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.075133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.075264 (+ 131us) server_negotiation.cc:407] Connection header received
0928 23:55:11.075354 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.075357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.075407 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.075491 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.076141 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.076850 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.077736 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.077973 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.078067 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.080808 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:11.080813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:11.080816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:11.080862 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:11.083018 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.083542 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.083546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.083547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.083640 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.083897 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.083899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.083900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.084089 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:11.084198 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.084515 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.084631 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 23:55:11.088044 28358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:11.091572 28358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:11.108630 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.102990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.20.88.2:46135)
0928 23:55:11.103352 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.103363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.103382 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:55:11.103447 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.103501 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.103673 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.103882 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.104149 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105480 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.107157 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.107951 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.108071 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.108219 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.108351 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.108464 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260928 23:55:11.108624 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.102560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:46135 (local address 127.0.0.1:48700)
0928 23:55:11.103024 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.103038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.103114 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.103859 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.103862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.103880 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.103993 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.104000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105622 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.105625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.107018 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.107030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.108094 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.108097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.108172 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.108193 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.108308 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.108466 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":188,"thread_start_us":94,"threads_started":1}
I20260928 23:55:11.109378 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.101761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35631 (local address 127.0.0.1:41128)
0928 23:55:11.102565 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.102622 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.102828 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.103606 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.103613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.103650 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.103981 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.103989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105472 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.105476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.106740 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.106751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.107845 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.107849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.107940 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.107975 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.109138 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.109229 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":241,"spinlock_wait_cycles":8448,"thread_start_us":108,"threads_started":1}
I20260928 23:55:11.109464 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.101916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.20.88.1:35631)
0928 23:55:11.102296 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.102303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.102712 (+ 409us) server_negotiation.cc:407] Connection header received
0928 23:55:11.103056 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.103064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.103303 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.103404 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.104114 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105329 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.106889 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.107708 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.107832 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.108015 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.109209 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.109324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":83,"threads_started":1}
I20260928 23:55:11.110208 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.101936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:41891 (local address 127.0.0.1:57348)
0928 23:55:11.102778 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.102792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.102880 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.103616 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.103619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.103648 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.103981 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.103989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105532 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.105539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.107018 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.107030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.109779 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.109782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.109887 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.109918 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.109990 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.110071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":150,"spinlock_wait_cycles":6400,"thread_start_us":110,"threads_started":1}
I20260928 23:55:11.110234 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.102614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.20.88.3:41891)
0928 23:55:11.103183 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.103198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.103221 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:55:11.103294 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.103299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.103477 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.103626 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.104338 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.105402 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.108100 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.109664 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.109782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.109961 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.110018 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.110104 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":116,"threads_started":1}
I20260928 23:55:11.112048 28636 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 (DEFAULT_TABLE table=test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.112879 28496 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 (DEFAULT_TABLE table=test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.113082 28766 tablet_service.cc:1467] Processing CreateTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 (DEFAULT_TABLE table=test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.113703 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f43a08c84e6e80fdaf9b1256ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.113914 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f43a08c84e6e80fdaf9b1256ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.114060 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7475f43a08c84e6e80fdaf9b1256ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.120772 28870 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Bootstrap starting.
I20260928 23:55:11.122455 28871 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Bootstrap starting.
I20260928 23:55:11.123294 28870 tablet_bootstrap.cc:654] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.123642 28872 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Bootstrap starting.
I20260928 23:55:11.124333 28870 log.cc:824] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:11.124933 28871 tablet_bootstrap.cc:654] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.125883 28871 log.cc:824] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:11.126104 28872 tablet_bootstrap.cc:654] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.126833 28870 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: No bootstrap required, opened a new log
I20260928 23:55:11.126960 28872 log.cc:824] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:11.127090 28870 ts_tablet_manager.cc:1397] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:55:11.128294 28871 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: No bootstrap required, opened a new log
I20260928 23:55:11.128513 28871 ts_tablet_manager.cc:1397] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:55:11.129264 28872 tablet_bootstrap.cc:492] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: No bootstrap required, opened a new log
I20260928 23:55:11.129465 28872 ts_tablet_manager.cc:1397] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:55:11.131281 28870 raft_consensus.cc:348] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.131621 28870 raft_consensus.cc:374] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.131678 28870 raft_consensus.cc:729] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.132200 28870 consensus_queue.cc:260] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.132989 28562 heartbeater.cc:503] Master 127.20.88.62:32837 was elected leader, sending a full tablet report...
I20260928 23:55:11.133387 28870 ts_tablet_manager.cc:1428] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 23:55:11.133499 28872 raft_consensus.cc:348] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.133770 28872 raft_consensus.cc:374] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.133829 28872 raft_consensus.cc:729] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.134044 28871 raft_consensus.cc:348] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.134333 28871 raft_consensus.cc:374] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.134397 28871 raft_consensus.cc:729] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.134320 28872 consensus_queue.cc:260] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.134825 28871 consensus_queue.cc:260] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.135037 28702 heartbeater.cc:503] Master 127.20.88.62:32837 was elected leader, sending a full tablet report...
I20260928 23:55:11.135401 28872 ts_tablet_manager.cc:1428] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:55:11.135834 28871 ts_tablet_manager.cc:1428] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:55:11.136046 28842 heartbeater.cc:503] Master 127.20.88.62:32837 was elected leader, sending a full tablet report...
W20260928 23:55:11.291353 28843 tablet.cc:2369] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:55:11.309955 28703 tablet.cc:2369] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:55:11.314883 28876 raft_consensus.cc:482] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:11.315145 28876 raft_consensus.cc:504] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.316453 28876 leader_election.cc:288] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.320645 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.316775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38733 (local address 127.20.88.2:46135)
0928 23:55:11.317010 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.317014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.317121 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:55:11.317279 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.317285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.317354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.317438 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.317753 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.318335 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.319528 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.319937 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.320069 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.320131 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.320416 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.320514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 23:55:11.320641 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.316854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:56219 (local address 127.20.88.3:41891)
0928 23:55:11.317050 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.317057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.317356 (+ 299us) server_negotiation.cc:407] Connection header received
0928 23:55:11.317512 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.317515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.317563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.317645 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.317927 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.318482 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.319537 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.319930 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.320052 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.320120 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.320409 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.320497 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 23:55:11.320667 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.316692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:46135 (local address 127.20.88.1:38733)
0928 23:55:11.316991 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.317007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.317159 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.317467 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.317471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.317490 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.317632 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.317641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.318447 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.318451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.319336 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.319345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.320092 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.320095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.320217 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.320286 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.320385 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.320493 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":10,"thread_start_us":97,"threads_started":1}
I20260928 23:55:11.320672 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.316772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:41891 (local address 127.20.88.1:56219)
0928 23:55:11.317255 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.317268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.317370 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.317668 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.317670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.317705 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.317822 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.317828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.318598 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.318600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.319384 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.319420 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.320071 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.320075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.320174 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.320282 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.320387 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.320513 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":108,"thread_start_us":100,"threads_started":1}
I20260928 23:55:11.321357 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f43a08c84e6e80fdaf9b1256ad82" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344" is_pre_election: true
I20260928 23:55:11.321346 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f43a08c84e6e80fdaf9b1256ad82" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f" is_pre_election: true
I20260928 23:55:11.321686 28796 raft_consensus.cc:2454] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:11.321688 28656 raft_consensus.cc:2454] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:11.322260 28449 leader_election.cc:302] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 7781005b6ab34b659e5e2c0682607344, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:11.322590 28876 raft_consensus.cc:2781] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:11.322702 28876 raft_consensus.cc:482] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:11.322784 28876 raft_consensus.cc:3037] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.323890 28876 raft_consensus.cc:504] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.324358 28876 leader_election.cc:288] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Requested vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.324872 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f43a08c84e6e80fdaf9b1256ad82" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f"
I20260928 23:55:11.325013 28656 raft_consensus.cc:3037] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.325066 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7475f43a08c84e6e80fdaf9b1256ad82" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344"
I20260928 23:55:11.325212 28796 raft_consensus.cc:3037] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:55:11.325515 28563 tablet.cc:2369] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:55:11.326475 28656 raft_consensus.cc:2454] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:11.326810 28449 leader_election.cc:302] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:11.326835 28796 raft_consensus.cc:2454] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:11.327018 28876 raft_consensus.cc:2781] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:11.327252 28876 raft_consensus.cc:686] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Becoming Leader. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Running, Role: LEADER
I20260928 23:55:11.327623 28876 consensus_queue.cc:237] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.331007 28358 catalog_manager.cc:5495] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef reported cstate change: term changed from 0 to 1, leader changed from <none> to 9104d649006240dc8899aaed93ed4cef (127.20.88.1). New cstate: current_term: 1 leader_uuid: "9104d649006240dc8899aaed93ed4cef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } health_report { overall_health: UNKNOWN } } }
I20260928 23:55:11.345700 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.341890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.20.88.1:35631)
0928 23:55:11.342016 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.342019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.342032 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:55:11.342111 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.342113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.342170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.342272 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:11.342554 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.343057 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.343995 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.344220 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.344383 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.344717 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.345196 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.345289 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.345477 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.345541 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:55:11.351974 28656 raft_consensus.cc:1261] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.351974 28796 raft_consensus.cc:1261] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.352782 28881 consensus_queue.cc:1035] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.352996 28876 consensus_queue.cc:1035] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.359591 28884 mvcc.cc:205] Tried to move back new op lower bound from 7334460257689612288 to 7334460257600790528. Current Snapshot: MvccSnapshot[applied={T|T < 7334460257689612288}]
I20260928 23:55:11.359916 28885 mvcc.cc:205] Tried to move back new op lower bound from 7334460257689612288 to 7334460257600790528. Current Snapshot: MvccSnapshot[applied={T|T < 7334460257689612288}]
I20260928 23:55:11.361678 28886 mvcc.cc:205] Tried to move back new op lower bound from 7334460257689612288 to 7334460257600790528. Current Snapshot: MvccSnapshot[applied={T|T < 7334460257689612288}]
I20260928 23:55:11.370437 28358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:11.370656 28358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:11.373718 28358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Sending DeleteTablet for 3 replicas of tablet 7475f43a08c84e6e80fdaf9b1256ad82
I20260928 23:55:11.374737 28766 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:57348
I20260928 23:55:11.374795 28496 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:41128
I20260928 23:55:11.374913 28636 tablet_service.cc:1514] Processing DeleteTablet for tablet 7475f43a08c84e6e80fdaf9b1256ad82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48700
I20260928 23:55:11.375344 28894 tablet_replica.cc:330] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: stopping tablet replica
I20260928 23:55:11.375726 28894 raft_consensus.cc:2229] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:11.376173 28894 raft_consensus.cc:2258] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.376339 28358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:11.376842 28358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:11.377774 28894 ts_tablet_manager.cc:1905] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.380345 28895 tablet_replica.cc:330] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: stopping tablet replica
I20260928 23:55:11.380748 28895 raft_consensus.cc:2229] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:11.381309 28893 tablet_replica.cc:330] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: stopping tablet replica
I20260928 23:55:11.381639 28893 raft_consensus.cc:2229] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:11.381994 28496 tablet_service.cc:1467] Processing CreateTablet for tablet a9a0c17117ae4066a35cf610c17615e1 (DEFAULT_TABLE table=test-table [id=7d71620152ba4798afc686af82727b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.382227 28766 tablet_service.cc:1467] Processing CreateTablet for tablet a9a0c17117ae4066a35cf610c17615e1 (DEFAULT_TABLE table=test-table [id=7d71620152ba4798afc686af82727b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.382344 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a0c17117ae4066a35cf610c17615e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.382376 28894 ts_tablet_manager.cc:1918] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:11.382494 28894 log.cc:1196] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/7475f43a08c84e6e80fdaf9b1256ad82
I20260928 23:55:11.382508 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a0c17117ae4066a35cf610c17615e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.382855 28894 ts_tablet_manager.cc:1939] T 7475f43a08c84e6e80fdaf9b1256ad82 P 9104d649006240dc8899aaed93ed4cef: Deleting consensus metadata
I20260928 23:55:11.384203 28893 raft_consensus.cc:2258] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.384510 28895 raft_consensus.cc:2258] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.384598 28870 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Bootstrap starting.
I20260928 23:55:11.385474 28871 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Bootstrap starting.
I20260928 23:55:11.385622 28870 tablet_bootstrap.cc:654] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.386070 28893 ts_tablet_manager.cc:1905] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.386381 28871 tablet_bootstrap.cc:654] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.386513 28345 catalog_manager.cc:4841] TS 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631): tablet 7475f43a08c84e6e80fdaf9b1256ad82 (table test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]) successfully deleted
I20260928 23:55:11.387219 28636 tablet_service.cc:1467] Processing CreateTablet for tablet a9a0c17117ae4066a35cf610c17615e1 (DEFAULT_TABLE table=test-table [id=7d71620152ba4798afc686af82727b6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.387502 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a0c17117ae4066a35cf610c17615e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.388675 28895 ts_tablet_manager.cc:1905] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.388741 28893 ts_tablet_manager.cc:1918] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:11.388835 28893 log.cc:1196] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/7475f43a08c84e6e80fdaf9b1256ad82
I20260928 23:55:11.389158 28893 ts_tablet_manager.cc:1939] T 7475f43a08c84e6e80fdaf9b1256ad82 P 7781005b6ab34b659e5e2c0682607344: Deleting consensus metadata
I20260928 23:55:11.390520 28872 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Bootstrap starting.
I20260928 23:55:11.390501 28343 catalog_manager.cc:4841] TS 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891): tablet 7475f43a08c84e6e80fdaf9b1256ad82 (table test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]) successfully deleted
I20260928 23:55:11.391224 28895 ts_tablet_manager.cc:1918] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:11.391357 28895 log.cc:1196] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/7475f43a08c84e6e80fdaf9b1256ad82
I20260928 23:55:11.391489 28872 tablet_bootstrap.cc:654] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.391701 28895 ts_tablet_manager.cc:1939] T 7475f43a08c84e6e80fdaf9b1256ad82 P 440b49a976fb4dab92bae5952b10404f: Deleting consensus metadata
I20260928 23:55:11.393078 28345 catalog_manager.cc:4841] TS 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135): tablet 7475f43a08c84e6e80fdaf9b1256ad82 (table test-table [id=b2c135716e5a41a3a3e5c9ef1f66ff72]) successfully deleted
I20260928 23:55:11.393762 28872 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: No bootstrap required, opened a new log
I20260928 23:55:11.393785 28870 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: No bootstrap required, opened a new log
I20260928 23:55:11.393844 28872 ts_tablet_manager.cc:1397] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:11.393878 28870 ts_tablet_manager.cc:1397] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260928 23:55:11.393921 28871 tablet_bootstrap.cc:492] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: No bootstrap required, opened a new log
I20260928 23:55:11.394016 28871 ts_tablet_manager.cc:1397] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 23:55:11.394294 28872 raft_consensus.cc:348] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394402 28872 raft_consensus.cc:374] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.394435 28872 raft_consensus.cc:729] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.394363 28870 raft_consensus.cc:348] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394497 28870 raft_consensus.cc:374] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.394436 28871 raft_consensus.cc:348] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394529 28870 raft_consensus.cc:729] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.394537 28871 raft_consensus.cc:374] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.394577 28871 raft_consensus.cc:729] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.394558 28872 consensus_queue.cc:260] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394690 28871 consensus_queue.cc:260] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394675 28870 consensus_queue.cc:260] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.394804 28872 ts_tablet_manager.cc:1428] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.394934 28870 ts_tablet_manager.cc:1428] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.394932 28871 ts_tablet_manager.cc:1428] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.542948 28881 raft_consensus.cc:482] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:11.543107 28881 raft_consensus.cc:504] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.543689 28881 leader_election.cc:288] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.544178 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a0c17117ae4066a35cf610c17615e1" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344" is_pre_election: true
I20260928 23:55:11.544284 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a0c17117ae4066a35cf610c17615e1" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f" is_pre_election: true
I20260928 23:55:11.544461 28656 raft_consensus.cc:2454] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:11.544473 28796 raft_consensus.cc:2454] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:11.544821 28449 leader_election.cc:302] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:11.545053 28881 raft_consensus.cc:2781] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:11.545118 28881 raft_consensus.cc:482] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:11.545169 28881 raft_consensus.cc:3037] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.546314 28881 raft_consensus.cc:504] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.546753 28881 leader_election.cc:288] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Requested vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.547163 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a0c17117ae4066a35cf610c17615e1" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f"
I20260928 23:55:11.547240 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a0c17117ae4066a35cf610c17615e1" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344"
I20260928 23:55:11.547326 28656 raft_consensus.cc:3037] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.547395 28796 raft_consensus.cc:3037] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.548624 28656 raft_consensus.cc:2454] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:11.548628 28796 raft_consensus.cc:2454] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:11.548944 28447 leader_election.cc:302] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7781005b6ab34b659e5e2c0682607344, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:11.549147 28881 raft_consensus.cc:2781] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:11.549275 28881 raft_consensus.cc:686] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Becoming Leader. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Running, Role: LEADER
I20260928 23:55:11.549453 28881 consensus_queue.cc:237] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.551267 28355 catalog_manager.cc:5495] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef reported cstate change: term changed from 0 to 1, leader changed from <none> to 9104d649006240dc8899aaed93ed4cef (127.20.88.1). New cstate: current_term: 1 leader_uuid: "9104d649006240dc8899aaed93ed4cef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639710, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.62@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:11.619643 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.610208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:32837 (local address 127.0.0.1:37138)
0928 23:55:11.610535 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.610549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.610657 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.610919 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.610921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.611407 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:11.611537 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.611545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.612572 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.612576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.613166 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.613175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.613675 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.613684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.613750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.614480 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:11.614494 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:11.616016 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:11.617958 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.617969 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.617975 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.618250 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.618611 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:11.618613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:11.618614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:11.618695 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:11.619100 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:11.619107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:11.619219 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.619314 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.619481 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260928 23:55:11.619681 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.610253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.20.88.62:32837)
0928 23:55:11.610557 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.610561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.610628 (+ 67us) server_negotiation.cc:407] Connection header received
0928 23:55:11.610741 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.610743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.610791 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.610879 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.611685 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.612342 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.613298 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.613544 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.613626 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.616156 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:11.616159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:11.616162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:11.616184 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:11.617823 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.618412 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.618416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.618417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.618508 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.618784 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.618786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.618788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.619000 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:11.619088 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.619405 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.619544 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260928 23:55:11.623817 28355 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:37138:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:55:11.624706 28355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:11.629923 28496 tablet_service.cc:1467] Processing CreateTablet for tablet e901e9e64f124991b36cf96948466cb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33dba8c867644411afb66744ab8c7df7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:55:11.630092 28636 tablet_service.cc:1467] Processing CreateTablet for tablet e901e9e64f124991b36cf96948466cb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33dba8c867644411afb66744ab8c7df7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:55:11.630092 28766 tablet_service.cc:1467] Processing CreateTablet for tablet e901e9e64f124991b36cf96948466cb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33dba8c867644411afb66744ab8c7df7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:55:11.630381 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901e9e64f124991b36cf96948466cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.630460 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901e9e64f124991b36cf96948466cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.630460 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901e9e64f124991b36cf96948466cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.633183 28870 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: Bootstrap starting.
I20260928 23:55:11.633378 28871 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: Bootstrap starting.
I20260928 23:55:11.633598 28872 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: Bootstrap starting.
I20260928 23:55:11.634001 28870 tablet_bootstrap.cc:654] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.634367 28871 tablet_bootstrap.cc:654] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.634541 28872 tablet_bootstrap.cc:654] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.635514 28870 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: No bootstrap required, opened a new log
I20260928 23:55:11.635613 28870 ts_tablet_manager.cc:1397] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 23:55:11.635864 28871 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: No bootstrap required, opened a new log
I20260928 23:55:11.635967 28871 ts_tablet_manager.cc:1397] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:11.636058 28870 raft_consensus.cc:348] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.636175 28870 raft_consensus.cc:374] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.636198 28870 raft_consensus.cc:729] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.636402 28870 consensus_queue.cc:260] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.636439 28871 raft_consensus.cc:348] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.636582 28871 raft_consensus.cc:374] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.636618 28871 raft_consensus.cc:729] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.636672 28870 ts_tablet_manager.cc:1428] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.636652 28881 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.636741 28871 consensus_queue.cc:260] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.636790 28881 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.636997 28871 ts_tablet_manager.cc:1428] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.637020 28878 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.637176 28878 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.637261 28872 tablet_bootstrap.cc:492] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: No bootstrap required, opened a new log
I20260928 23:55:11.637342 28872 ts_tablet_manager.cc:1397] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:55:11.637809 28872 raft_consensus.cc:348] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.637933 28872 raft_consensus.cc:374] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.637972 28872 raft_consensus.cc:729] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.638095 28872 consensus_queue.cc:260] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.638417 28872 ts_tablet_manager.cc:1428] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:11.638677 28877 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.638810 28877 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.742869 28877 raft_consensus.cc:482] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:11.743037 28877 raft_consensus.cc:504] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.744165 28877 leader_election.cc:288] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.748773 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.744543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:56683 (local address 127.20.88.1:35631)
0928 23:55:11.744706 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.744710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.745073 (+ 363us) server_negotiation.cc:407] Connection header received
0928 23:55:11.745222 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.745225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.745306 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.745438 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.745726 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.746203 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.747325 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.747951 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.748077 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.748147 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.748549 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.748632 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:55:11.748773 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.744618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:53021 (local address 127.20.88.3:41891)
0928 23:55:11.744744 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.744748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.745002 (+ 254us) server_negotiation.cc:407] Connection header received
0928 23:55:11.745159 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.745165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.745219 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.745285 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:11.745615 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.746161 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.747386 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.747977 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.748077 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.748144 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.748549 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.748632 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:55:11.748845 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.744543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35631 (local address 127.20.88.2:56683)
0928 23:55:11.744922 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.744941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.745098 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.745458 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.745460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.745474 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.745609 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.745616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.746353 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.746356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.747189 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.747198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.748138 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.748142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.748355 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.748411 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.748525 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.748653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":111,"thread_start_us":101,"threads_started":1}
I20260928 23:55:11.749295 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.744492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:41891 (local address 127.20.88.2:53021)
0928 23:55:11.744889 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.744903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.745010 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.745321 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.745326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.745347 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:11.745512 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.745524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.746280 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.746287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.747229 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.747242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.748138 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.748142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.748350 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.748418 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.748505 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.748657 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":122,"thread_start_us":149,"threads_started":1}
I20260928 23:55:11.749477 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901e9e64f124991b36cf96948466cb9" candidate_uuid: "440b49a976fb4dab92bae5952b10404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9104d649006240dc8899aaed93ed4cef" is_pre_election: true
I20260928 23:55:11.749787 28516 raft_consensus.cc:2454] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440b49a976fb4dab92bae5952b10404f in term 0.
I20260928 23:55:11.749838 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901e9e64f124991b36cf96948466cb9" candidate_uuid: "440b49a976fb4dab92bae5952b10404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344" is_pre_election: true
I20260928 23:55:11.749987 28796 raft_consensus.cc:2454] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440b49a976fb4dab92bae5952b10404f in term 0.
I20260928 23:55:11.750397 28587 leader_election.cc:302] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 7781005b6ab34b659e5e2c0682607344, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:11.750684 28877 raft_consensus.cc:2781] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:11.750747 28877 raft_consensus.cc:482] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:11.750945 28877 raft_consensus.cc:3037] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.752105 28877 raft_consensus.cc:504] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.752557 28877 leader_election.cc:288] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [CANDIDATE]: Term 1 election: Requested vote from peers 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:11.752970 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901e9e64f124991b36cf96948466cb9" candidate_uuid: "440b49a976fb4dab92bae5952b10404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344"
I20260928 23:55:11.752993 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901e9e64f124991b36cf96948466cb9" candidate_uuid: "440b49a976fb4dab92bae5952b10404f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9104d649006240dc8899aaed93ed4cef"
I20260928 23:55:11.753090 28796 raft_consensus.cc:3037] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.753163 28516 raft_consensus.cc:3037] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:11.754076 28796 raft_consensus.cc:2454] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440b49a976fb4dab92bae5952b10404f in term 1.
I20260928 23:55:11.754237 28516 raft_consensus.cc:2454] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440b49a976fb4dab92bae5952b10404f in term 1.
I20260928 23:55:11.754437 28587 leader_election.cc:302] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 7781005b6ab34b659e5e2c0682607344; no voters:
I20260928 23:55:11.754626 28877 raft_consensus.cc:2781] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:11.754823 28877 raft_consensus.cc:686] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [term 1 LEADER]: Becoming Leader. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Running, Role: LEADER
I20260928 23:55:11.755103 28877 consensus_queue.cc:237] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.757303 28912 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: TxnStatusTablet state changed. Reason: New leader 440b49a976fb4dab92bae5952b10404f. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.757441 28912 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:55:11.757737 28914 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:55:11.757925 28914 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:55:11.758081 28355 catalog_manager.cc:5495] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f reported cstate change: term changed from 0 to 1, leader changed from <none> to 440b49a976fb4dab92bae5952b10404f (127.20.88.2). New cstate: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } health_report { overall_health: UNKNOWN } } }
I20260928 23:55:11.799176 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.794532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:46135 (local address 127.0.0.1:48714)
0928 23:55:11.794708 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.794726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.794847 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.795165 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.795167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.795176 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:55:11.795272 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.795276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.796344 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.796347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.797223 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.797235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.797842 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.797845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.797927 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.797981 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.798880 (+ 899us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.798895 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.798965 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.799051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 23:55:11.799220 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.794708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.20.88.2:46135)
0928 23:55:11.794861 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.794866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.794883 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:55:11.794965 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.794968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.795044 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.795150 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:11.795380 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.796191 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.797358 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.797716 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.797824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.798136 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.798750 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.798871 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.798986 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.799058 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:55:11.802626 28796 raft_consensus.cc:1261] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Refusing update from remote peer 440b49a976fb4dab92bae5952b10404f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.802820 28516 raft_consensus.cc:1261] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Refusing update from remote peer 440b49a976fb4dab92bae5952b10404f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.803179 28877 consensus_queue.cc:1035] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.803426 28912 consensus_queue.cc:1035] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.805284 28878 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: TxnStatusTablet state changed. Reason: New leader 440b49a976fb4dab92bae5952b10404f. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.805423 28878 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.805353 28881 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: TxnStatusTablet state changed. Reason: New leader 440b49a976fb4dab92bae5952b10404f. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.805470 28881 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.807032 28912 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.807161 28912 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:55:11.811580 28912 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.811717 28912 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 440b49a976fb4dab92bae5952b10404f: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:55:11.813393 28878 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.813530 28878 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 7781005b6ab34b659e5e2c0682607344: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.813737 28881 tablet_replica.cc:439] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "440b49a976fb4dab92bae5952b10404f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } } }
I20260928 23:55:11.813853 28881 tablet_replica.cc:442] T e901e9e64f124991b36cf96948466cb9 P 9104d649006240dc8899aaed93ed4cef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:55:11.828822 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.823844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38541 (local address 127.20.88.2:46135)
0928 23:55:11.823963 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.823969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.823984 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:11.824042 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.824045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.824101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.824192 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:11.824501 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.825431 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.827373 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.827776 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.827900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.828293 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.828395 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.828479 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.828600 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.828666 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:55:11.828828 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.823488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:46135 (local address 127.20.88.1:38541)
0928 23:55:11.823761 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.823779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.823885 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.824213 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.824216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.824225 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:55:11.824376 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.824382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.825550 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.825554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.826629 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.826641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.827917 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.827921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.828016 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.828079 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.828492 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.828498 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.828565 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.828666 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260928 23:55:11.837522 28656 raft_consensus.cc:1261] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.838020 28876 consensus_queue.cc:1035] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.838176 28796 raft_consensus.cc:1261] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:11.838583 28876 consensus_queue.cc:1035] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:11.860781 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.856917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35631 (local address 127.20.88.2:39959)
0928 23:55:11.857280 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:11.857293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:11.857383 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:11.857639 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:11.857642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:11.857671 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:55:11.857805 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.857814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.858575 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.858579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.859478 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:11.859495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.859967 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.859970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:11.860074 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.860122 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.860492 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.860500 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:55:11.860564 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:55:11.860646 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":144,"threads_started":1}
I20260928 23:55:11.860781 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.857058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:39959 (local address 127.20.88.1:35631)
0928 23:55:11.857212 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.857217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.857371 (+ 154us) server_negotiation.cc:407] Connection header received
0928 23:55:11.857462 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.857466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.857538 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.857657 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:11.857926 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.858453 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.859605 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.859846 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.859963 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.860306 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:11.860394 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:11.860479 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.860575 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.860633 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:55:11.883980 28358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:11.884140 28358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:11.886566 28358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Sending DeleteTablet for 3 replicas of tablet a9a0c17117ae4066a35cf610c17615e1
I20260928 23:55:11.887320 28766 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a0c17117ae4066a35cf610c17615e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:57348
I20260928 23:55:11.887490 28496 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a0c17117ae4066a35cf610c17615e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:41128
I20260928 23:55:11.887564 28893 tablet_replica.cc:330] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: stopping tablet replica
I20260928 23:55:11.887714 28893 raft_consensus.cc:2229] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:11.887861 28893 raft_consensus.cc:2258] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.888864 28893 ts_tablet_manager.cc:1905] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.889572 28636 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a0c17117ae4066a35cf610c17615e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:11 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48700
I20260928 23:55:11.889782 28895 tablet_replica.cc:330] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: stopping tablet replica
I20260928 23:55:11.889895 28895 raft_consensus.cc:2229] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:11.890027 28895 raft_consensus.cc:2258] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.891114 28895 ts_tablet_manager.cc:1905] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.891953 28893 ts_tablet_manager.cc:1918] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:55:11.892021 28893 log.cc:1196] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/a9a0c17117ae4066a35cf610c17615e1
I20260928 23:55:11.892216 28893 ts_tablet_manager.cc:1939] T a9a0c17117ae4066a35cf610c17615e1 P 7781005b6ab34b659e5e2c0682607344: Deleting consensus metadata
I20260928 23:55:11.893327 28343 catalog_manager.cc:4841] TS 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891): tablet a9a0c17117ae4066a35cf610c17615e1 (table test-table [id=7d71620152ba4798afc686af82727b6e]) successfully deleted
I20260928 23:55:11.894053 28931 tablet_replica.cc:330] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: stopping tablet replica
I20260928 23:55:11.894222 28931 raft_consensus.cc:2229] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:11.894472 28931 raft_consensus.cc:2258] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:11.895169 28895 ts_tablet_manager.cc:1918] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:55:11.895282 28931 ts_tablet_manager.cc:1905] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:11.895321 28895 log.cc:1196] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/a9a0c17117ae4066a35cf610c17615e1
I20260928 23:55:11.895727 28895 ts_tablet_manager.cc:1939] T a9a0c17117ae4066a35cf610c17615e1 P 440b49a976fb4dab92bae5952b10404f: Deleting consensus metadata
I20260928 23:55:11.896660 28345 catalog_manager.cc:4841] TS 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135): tablet a9a0c17117ae4066a35cf610c17615e1 (table test-table [id=7d71620152ba4798afc686af82727b6e]) successfully deleted
I20260928 23:55:11.898638 28931 ts_tablet_manager.cc:1918] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:55:11.898723 28931 log.cc:1196] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/a9a0c17117ae4066a35cf610c17615e1
I20260928 23:55:11.899024 28931 ts_tablet_manager.cc:1939] T a9a0c17117ae4066a35cf610c17615e1 P 9104d649006240dc8899aaed93ed4cef: Deleting consensus metadata
I20260928 23:55:11.899896 28345 catalog_manager.cc:4841] TS 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631): tablet a9a0c17117ae4066a35cf610c17615e1 (table test-table [id=7d71620152ba4798afc686af82727b6e]) successfully deleted
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:55:11.919823 28906 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:11.912366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.20.88.62:32837)
0928 23:55:11.912555 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.912561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.912656 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:55:11.912793 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.912797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.912865 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.912975 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.913959 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.914888 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.915964 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.916415 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.916564 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.919663 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37152: BlockingRecv error: recv got EOF from 127.0.0.1:37152 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 23:55:11 dist-test-slave-wvpw krb5kdc[28314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM
I20260928 23:55:11.931391 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:11.921631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.20.88.62:32837)
0928 23:55:11.921766 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:11.921770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:11.921853 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:55:11.921959 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:11.921963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:11.922030 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:11.922136 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:11.922713 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.923579 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.924521 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:11.924903 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:11.925036 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:11.927570 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:11.927575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:11.927578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:11.927611 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:11.929443 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.929875 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.929879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.929907 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.930017 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:11.930347 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:11.930351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:11.930353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:11.930681 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:11.930836 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:11.931060 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 23:55:11.931210 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:55:11.934276 28358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:11.934823 28358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:11.940117 28496 tablet_service.cc:1467] Processing CreateTablet for tablet fec55b132f2d419590f56aa3cbec17e8 (DEFAULT_TABLE table=test-table [id=17db5dd549f7436fa6e15ce92a61e02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.940145 28636 tablet_service.cc:1467] Processing CreateTablet for tablet fec55b132f2d419590f56aa3cbec17e8 (DEFAULT_TABLE table=test-table [id=17db5dd549f7436fa6e15ce92a61e02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.940501 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec55b132f2d419590f56aa3cbec17e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.940500 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec55b132f2d419590f56aa3cbec17e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.943024 28870 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Bootstrap starting.
I20260928 23:55:11.943035 28872 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Bootstrap starting.
I20260928 23:55:11.944048 28872 tablet_bootstrap.cc:654] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.944130 28870 tablet_bootstrap.cc:654] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.944974 28766 tablet_service.cc:1467] Processing CreateTablet for tablet fec55b132f2d419590f56aa3cbec17e8 (DEFAULT_TABLE table=test-table [id=17db5dd549f7436fa6e15ce92a61e02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:11.945294 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec55b132f2d419590f56aa3cbec17e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:11.945531 28870 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: No bootstrap required, opened a new log
I20260928 23:55:11.945629 28870 ts_tablet_manager.cc:1397] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 23:55:11.946053 28870 raft_consensus.cc:348] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.946142 28872 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: No bootstrap required, opened a new log
I20260928 23:55:11.946170 28870 raft_consensus.cc:374] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.946204 28870 raft_consensus.cc:729] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.946231 28872 ts_tablet_manager.cc:1397] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 23:55:11.946333 28870 consensus_queue.cc:260] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.946635 28870 ts_tablet_manager.cc:1428] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:55:11.946667 28872 raft_consensus.cc:348] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.946776 28872 raft_consensus.cc:374] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.946807 28872 raft_consensus.cc:729] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.946933 28872 consensus_queue.cc:260] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.947190 28872 ts_tablet_manager.cc:1428] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:55:11.948505 28871 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Bootstrap starting.
I20260928 23:55:11.949546 28871 tablet_bootstrap.cc:654] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:11.950909 28871 tablet_bootstrap.cc:492] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: No bootstrap required, opened a new log
I20260928 23:55:11.950986 28871 ts_tablet_manager.cc:1397] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:11.951490 28871 raft_consensus.cc:348] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.951624 28871 raft_consensus.cc:374] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:11.951660 28871 raft_consensus.cc:729] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Initialized, Role: FOLLOWER
I20260928 23:55:11.951794 28871 consensus_queue.cc:260] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:11.952128 28871 ts_tablet_manager.cc:1428] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:12.087940 28878 raft_consensus.cc:482] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:12.088119 28878 raft_consensus.cc:504] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.089411 28878 leader_election.cc:288] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631), 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135)
I20260928 23:55:12.094600 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.089685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35631 (local address 127.20.88.3:51795)
0928 23:55:12.090170 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:12.090188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:12.090312 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:12.090723 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:12.090725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:12.090742 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:12.090827 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:12.090831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.092203 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.092207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:12.093244 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:12.093253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.094097 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.094101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:12.094195 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.094223 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:55:12.094307 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:55:12.094388 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":88,"threads_started":1}
I20260928 23:55:12.094662 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.089832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:51795 (local address 127.20.88.1:35631)
0928 23:55:12.090105 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:12.090109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:12.090305 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:55:12.090472 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:12.090475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:12.090536 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:12.090592 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:12.091186 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.092080 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.093413 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.093969 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.094054 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.094115 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:12.094426 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 23:55:12.094529 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260928 23:55:12.095105 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.090446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:34709 (local address 127.20.88.2:46135)
0928 23:55:12.090575 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:12.090580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:12.091167 (+ 587us) server_negotiation.cc:407] Connection header received
0928 23:55:12.091410 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:12.091412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:12.091459 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:12.091563 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:12.091818 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.092553 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.093854 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.094450 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.094553 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.094621 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:12.094884 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:55:12.094972 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:55:12.095144 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec55b132f2d419590f56aa3cbec17e8" candidate_uuid: "7781005b6ab34b659e5e2c0682607344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9104d649006240dc8899aaed93ed4cef" is_pre_election: true
I20260928 23:55:12.095358 28516 raft_consensus.cc:2454] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7781005b6ab34b659e5e2c0682607344 in term 0.
I20260928 23:55:12.095541 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.090247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:46135 (local address 127.20.88.3:34709)
0928 23:55:12.090614 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:12.090636 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:12.091216 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:12.091575 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:12.091578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:12.091593 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:12.091689 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:12.091695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.092724 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.092726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:12.093674 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:12.093683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.094582 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.094586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:12.094682 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.094714 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:55:12.095268 (+ 554us) client_negotiation.cc:240] Negotiation successful
0928 23:55:12.095355 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260928 23:55:12.095851 28729 leader_election.cc:302] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7781005b6ab34b659e5e2c0682607344, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:12.096160 28878 raft_consensus.cc:2781] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:12.096261 28878 raft_consensus.cc:482] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:12.096318 28878 raft_consensus.cc:3037] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.096602 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec55b132f2d419590f56aa3cbec17e8" candidate_uuid: "7781005b6ab34b659e5e2c0682607344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f" is_pre_election: true
I20260928 23:55:12.096738 28656 raft_consensus.cc:2454] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7781005b6ab34b659e5e2c0682607344 in term 0.
I20260928 23:55:12.097586 28878 raft_consensus.cc:504] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.098006 28878 leader_election.cc:288] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [CANDIDATE]: Term 1 election: Requested vote from peers 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631), 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135)
I20260928 23:55:12.098512 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec55b132f2d419590f56aa3cbec17e8" candidate_uuid: "7781005b6ab34b659e5e2c0682607344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f"
I20260928 23:55:12.098554 28516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec55b132f2d419590f56aa3cbec17e8" candidate_uuid: "7781005b6ab34b659e5e2c0682607344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9104d649006240dc8899aaed93ed4cef"
I20260928 23:55:12.098666 28656 raft_consensus.cc:3037] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.098703 28516 raft_consensus.cc:3037] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.099696 28656 raft_consensus.cc:2454] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7781005b6ab34b659e5e2c0682607344 in term 1.
I20260928 23:55:12.099725 28516 raft_consensus.cc:2454] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7781005b6ab34b659e5e2c0682607344 in term 1.
I20260928 23:55:12.100062 28729 leader_election.cc:302] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 7781005b6ab34b659e5e2c0682607344; no voters:
I20260928 23:55:12.100287 28878 raft_consensus.cc:2781] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:12.100543 28878 raft_consensus.cc:686] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 1 LEADER]: Becoming Leader. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Running, Role: LEADER
I20260928 23:55:12.100816 28878 consensus_queue.cc:237] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.103593 28355 catalog_manager.cc:5495] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7781005b6ab34b659e5e2c0682607344 (127.20.88.3). New cstate: current_term: 1 leader_uuid: "7781005b6ab34b659e5e2c0682607344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } health_report { overall_health: HEALTHY } } }
I20260928 23:55:12.146066 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.142167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.20.88.3:41891)
0928 23:55:12.142319 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:12.142323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:12.142337 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:12.142471 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:12.142473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:12.142527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:12.142628 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:12.142896 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.143424 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.144415 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.144719 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.144806 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.145091 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:12.145658 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:12.145748 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:12.145827 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:55:12.145877 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:55:12.149660 28516 raft_consensus.cc:1261] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Refusing update from remote peer 7781005b6ab34b659e5e2c0682607344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:12.149675 28656 raft_consensus.cc:1261] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Refusing update from remote peer 7781005b6ab34b659e5e2c0682607344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:12.150306 28878 consensus_queue.cc:1035] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:12.150607 28949 consensus_queue.cc:1035] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:12.162837 28355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:12.163039 28355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:12.166281 28355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Sending DeleteTablet for 3 replicas of tablet fec55b132f2d419590f56aa3cbec17e8
I20260928 23:55:12.167047 28496 tablet_service.cc:1514] Processing DeleteTablet for tablet fec55b132f2d419590f56aa3cbec17e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:41128
I20260928 23:55:12.167114 28766 tablet_service.cc:1514] Processing DeleteTablet for tablet fec55b132f2d419590f56aa3cbec17e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:57348
I20260928 23:55:12.167109 28636 tablet_service.cc:1514] Processing DeleteTablet for tablet fec55b132f2d419590f56aa3cbec17e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48700
I20260928 23:55:12.167259 28931 tablet_replica.cc:330] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: stopping tablet replica
I20260928 23:55:12.167307 28893 tablet_replica.cc:330] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: stopping tablet replica
I20260928 23:55:12.167388 28931 raft_consensus.cc:2229] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:12.167430 28893 raft_consensus.cc:2229] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:12.167551 28931 raft_consensus.cc:2258] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.167740 28895 tablet_replica.cc:330] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: stopping tablet replica
I20260928 23:55:12.167763 28893 raft_consensus.cc:2258] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.167865 28895 raft_consensus.cc:2229] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:12.167999 28895 raft_consensus.cc:2258] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.168614 28893 ts_tablet_manager.cc:1905] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:12.168642 28355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:12.169152 28355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:12.170325 28895 ts_tablet_manager.cc:1905] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:12.171346 28893 ts_tablet_manager.cc:1918] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:12.171452 28893 log.cc:1196] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/fec55b132f2d419590f56aa3cbec17e8
I20260928 23:55:12.171700 28893 ts_tablet_manager.cc:1939] T fec55b132f2d419590f56aa3cbec17e8 P 7781005b6ab34b659e5e2c0682607344: Deleting consensus metadata
I20260928 23:55:12.172678 28343 catalog_manager.cc:4841] TS 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891): tablet fec55b132f2d419590f56aa3cbec17e8 (table test-table [id=17db5dd549f7436fa6e15ce92a61e02b]) successfully deleted
I20260928 23:55:12.172807 28895 ts_tablet_manager.cc:1918] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:12.172901 28895 log.cc:1196] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/fec55b132f2d419590f56aa3cbec17e8
I20260928 23:55:12.173163 28895 ts_tablet_manager.cc:1939] T fec55b132f2d419590f56aa3cbec17e8 P 440b49a976fb4dab92bae5952b10404f: Deleting consensus metadata
I20260928 23:55:12.173846 28345 catalog_manager.cc:4841] TS 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135): tablet fec55b132f2d419590f56aa3cbec17e8 (table test-table [id=17db5dd549f7436fa6e15ce92a61e02b]) successfully deleted
I20260928 23:55:12.174386 28931 ts_tablet_manager.cc:1905] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:12.177294 28931 ts_tablet_manager.cc:1918] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:12.177409 28931 log.cc:1196] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/fec55b132f2d419590f56aa3cbec17e8
I20260928 23:55:12.177693 28931 ts_tablet_manager.cc:1939] T fec55b132f2d419590f56aa3cbec17e8 P 9104d649006240dc8899aaed93ed4cef: Deleting consensus metadata
I20260928 23:55:12.177834 28766 tablet_service.cc:1467] Processing CreateTablet for tablet 7faebb97b5324b539d968e5355cd9d4c (DEFAULT_TABLE table=test-table [id=37fdfdeedcdc409eb4f4b97e48f781c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:12.177913 28636 tablet_service.cc:1467] Processing CreateTablet for tablet 7faebb97b5324b539d968e5355cd9d4c (DEFAULT_TABLE table=test-table [id=37fdfdeedcdc409eb4f4b97e48f781c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:12.178160 28766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faebb97b5324b539d968e5355cd9d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:12.178171 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faebb97b5324b539d968e5355cd9d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:12.180105 28872 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Bootstrap starting.
I20260928 23:55:12.180608 28871 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Bootstrap starting.
I20260928 23:55:12.180583 28345 catalog_manager.cc:4841] TS 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631): tablet fec55b132f2d419590f56aa3cbec17e8 (table test-table [id=17db5dd549f7436fa6e15ce92a61e02b]) successfully deleted
I20260928 23:55:12.181169 28872 tablet_bootstrap.cc:654] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:12.181550 28871 tablet_bootstrap.cc:654] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:12.182349 28496 tablet_service.cc:1467] Processing CreateTablet for tablet 7faebb97b5324b539d968e5355cd9d4c (DEFAULT_TABLE table=test-table [id=37fdfdeedcdc409eb4f4b97e48f781c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:12.182425 28872 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: No bootstrap required, opened a new log
I20260928 23:55:12.182508 28872 ts_tablet_manager.cc:1397] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 23:55:12.182616 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faebb97b5324b539d968e5355cd9d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:12.182849 28872 raft_consensus.cc:348] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.182940 28872 raft_consensus.cc:374] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:12.182960 28872 raft_consensus.cc:729] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440b49a976fb4dab92bae5952b10404f, State: Initialized, Role: FOLLOWER
I20260928 23:55:12.183113 28872 consensus_queue.cc:260] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.183429 28872 ts_tablet_manager.cc:1428] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:12.185429 28871 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: No bootstrap required, opened a new log
I20260928 23:55:12.185526 28871 ts_tablet_manager.cc:1397] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 23:55:12.185990 28871 raft_consensus.cc:348] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.186106 28871 raft_consensus.cc:374] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:12.186136 28871 raft_consensus.cc:729] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7781005b6ab34b659e5e2c0682607344, State: Initialized, Role: FOLLOWER
I20260928 23:55:12.186260 28871 consensus_queue.cc:260] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.186524 28871 ts_tablet_manager.cc:1428] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:55:12.187855 28870 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: Bootstrap starting.
I20260928 23:55:12.188964 28870 tablet_bootstrap.cc:654] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:12.190320 28870 tablet_bootstrap.cc:492] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: No bootstrap required, opened a new log
I20260928 23:55:12.190428 28870 ts_tablet_manager.cc:1397] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:12.190850 28870 raft_consensus.cc:348] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.190982 28870 raft_consensus.cc:374] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:12.191028 28870 raft_consensus.cc:729] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Initialized, Role: FOLLOWER
I20260928 23:55:12.191164 28870 consensus_queue.cc:260] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.191481 28870 ts_tablet_manager.cc:1428] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:55:12.243237 28928 raft_consensus.cc:482] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:12.243417 28928 raft_consensus.cc:504] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.243983 28928 leader_election.cc:288] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:12.244496 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faebb97b5324b539d968e5355cd9d4c" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f" is_pre_election: true
I20260928 23:55:12.244544 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faebb97b5324b539d968e5355cd9d4c" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344" is_pre_election: true
I20260928 23:55:12.244671 28656 raft_consensus.cc:2454] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:12.244722 28796 raft_consensus.cc:2454] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 0.
I20260928 23:55:12.245059 28449 leader_election.cc:302] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440b49a976fb4dab92bae5952b10404f, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:12.245239 28928 raft_consensus.cc:2781] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:12.245291 28928 raft_consensus.cc:482] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:12.245328 28928 raft_consensus.cc:3037] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.246608 28928 raft_consensus.cc:504] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.247005 28928 leader_election.cc:288] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Requested vote from peers 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135), 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891)
I20260928 23:55:12.247399 28656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faebb97b5324b539d968e5355cd9d4c" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440b49a976fb4dab92bae5952b10404f"
I20260928 23:55:12.247445 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faebb97b5324b539d968e5355cd9d4c" candidate_uuid: "9104d649006240dc8899aaed93ed4cef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7781005b6ab34b659e5e2c0682607344"
I20260928 23:55:12.247535 28656 raft_consensus.cc:3037] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.247599 28796 raft_consensus.cc:3037] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:12.248638 28796 raft_consensus.cc:2454] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:12.248688 28656 raft_consensus.cc:2454] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9104d649006240dc8899aaed93ed4cef in term 1.
I20260928 23:55:12.248956 28447 leader_election.cc:302] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7781005b6ab34b659e5e2c0682607344, 9104d649006240dc8899aaed93ed4cef; no voters:
I20260928 23:55:12.249145 28928 raft_consensus.cc:2781] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:12.249294 28928 raft_consensus.cc:686] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Becoming Leader. State: Replica: 9104d649006240dc8899aaed93ed4cef, State: Running, Role: LEADER
I20260928 23:55:12.249476 28928 consensus_queue.cc:237] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } }
I20260928 23:55:12.251173 28357 catalog_manager.cc:5495] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef reported cstate change: term changed from 0 to 1, leader changed from <none> to 9104d649006240dc8899aaed93ed4cef (127.20.88.1). New cstate: current_term: 1 leader_uuid: "9104d649006240dc8899aaed93ed4cef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9104d649006240dc8899aaed93ed4cef" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 } health_report { overall_health: UNKNOWN } } }
I20260928 23:55:12.281478 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.277677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.20.88.1:35631)
0928 23:55:12.278035 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:12.278038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:12.278049 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:55:12.278089 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:12.278091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:12.278131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:12.278216 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:12.278484 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.279002 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.279884 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:12.280176 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:12.280336 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:12.280636 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:12.280724 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:12.280808 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:12.281221 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 23:55:12.281274 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260928 23:55:12.288730 28656 raft_consensus.cc:1261] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:12.288726 28796 raft_consensus.cc:1261] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Refusing update from remote peer 9104d649006240dc8899aaed93ed4cef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:12.289129 28876 consensus_queue.cc:1035] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "440b49a976fb4dab92bae5952b10404f" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:12.289263 28928 consensus_queue.cc:1035] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7781005b6ab34b659e5e2c0682607344" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:12.327771 28355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:12.327962 28355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:12.330165 28355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787c7906e9d14a0b8c8248de978be6a2: Sending DeleteTablet for 3 replicas of tablet 7faebb97b5324b539d968e5355cd9d4c
I20260928 23:55:12.330801 28496 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faebb97b5324b539d968e5355cd9d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:41128
I20260928 23:55:12.330833 28636 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faebb97b5324b539d968e5355cd9d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:48700
I20260928 23:55:12.330861 28766 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faebb97b5324b539d968e5355cd9d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:12 UTC) from {username='oryx', principal='oryx/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:57348
I20260928 23:55:12.331001 28895 tablet_replica.cc:330] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: stopping tablet replica
I20260928 23:55:12.331040 28931 tablet_replica.cc:330] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef: stopping tablet replica
I20260928 23:55:12.331135 28931 raft_consensus.cc:2229] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:12.331120 28895 raft_consensus.cc:2229] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:12.331247 28895 raft_consensus.cc:2258] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.331341 28931 raft_consensus.cc:2258] T 7faebb97b5324b539d968e5355cd9d4c P 9104d649006240dc8899aaed93ed4cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.331409 28893 tablet_replica.cc:330] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: stopping tablet replica
I20260928 23:55:12.331528 28893 raft_consensus.cc:2229] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:12.331652 28893 raft_consensus.cc:2258] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:12.332223 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 28431
I20260928 23:55:12.332518 28895 ts_tablet_manager.cc:1905] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:12.335242 28895 ts_tablet_manager.cc:1918] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:55:12.335345 28895 log.cc:1196] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/7faebb97b5324b539d968e5355cd9d4c
I20260928 23:55:12.335592 28895 ts_tablet_manager.cc:1939] T 7faebb97b5324b539d968e5355cd9d4c P 440b49a976fb4dab92bae5952b10404f: Deleting consensus metadata
I20260928 23:55:12.336588 28345 catalog_manager.cc:4841] TS 440b49a976fb4dab92bae5952b10404f (127.20.88.2:46135): tablet 7faebb97b5324b539d968e5355cd9d4c (table test-table [id=37fdfdeedcdc409eb4f4b97e48f781c3]) successfully deleted
I20260928 23:55:12.341495 28893 ts_tablet_manager.cc:1905] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:55:12.343431 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:35631 (error 108)
I20260928 23:55:12.344170 28893 ts_tablet_manager.cc:1918] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:55:12.344287 28893 log.cc:1196] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/7faebb97b5324b539d968e5355cd9d4c
I20260928 23:55:12.344550 28893 ts_tablet_manager.cc:1939] T 7faebb97b5324b539d968e5355cd9d4c P 7781005b6ab34b659e5e2c0682607344: Deleting consensus metadata
I20260928 23:55:12.344753 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 28571
I20260928 23:55:12.346949 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:12.344281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35631 (local address 127.0.0.1:41160)
0928 23:55:12.346734 (+ 2453us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:12.346808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2372,"thread_start_us":54,"threads_started":1}
W20260928 23:55:12.347111 28345 catalog_manager.cc:4584] TS 9104d649006240dc8899aaed93ed4cef (127.20.88.1:35631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7faebb97b5324b539d968e5355cd9d4c: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35631: connect: Connection refused (error 111)
I20260928 23:55:12.347870 28343 catalog_manager.cc:4841] TS 7781005b6ab34b659e5e2c0682607344 (127.20.88.3:41891): tablet 7faebb97b5324b539d968e5355cd9d4c (table test-table [id=37fdfdeedcdc409eb4f4b97e48f781c3]) successfully deleted
I20260928 23:55:12.354022 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 28711
I20260928 23:55:12.361265 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 28327
2026-09-28T23:55:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3486 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:12 dist-test-slave-wvpw krb5kdc[28968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:12 dist-test-slave-wvpw krb5kdc[28968](info): set up 2 sockets
Sep 28 23:55:12 dist-test-slave-wvpw krb5kdc[28968](info): commencing operation
krb5kdc: starting...
W20260928 23:55:13.413652 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:13 dist-test-slave-wvpw krb5kdc[28968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:55:13.548962 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:38439
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:13.672622 28981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:13.672937 28981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:13.672979 28981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:13.677482 28981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:13.677562 28981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:13.677584 28981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:13.677603 28981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:13.677618 28981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:13.683399 28981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:13.684644 28981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:13.685827 28981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:13.695210 28989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:13.695472 28981 server_base.cc:1034] running on GCE node
W20260928 23:55:13.695261 28986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:13.695250 28987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:13.696493 28981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:13.697718 28981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:13.698902 28981 hybrid_clock.cc:648] HybridClock initialized: now 1790639713698868 us; error 53 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T23:55:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1345 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:13 dist-test-slave-wvpw krb5kdc[28994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:13 dist-test-slave-wvpw krb5kdc[28994](info): set up 2 sockets
Sep 28 23:55:13 dist-test-slave-wvpw krb5kdc[28994](info): commencing operation
krb5kdc: starting...
W20260928 23:55:14.765609 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:14 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:55:14.901016 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42667
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:15.016208 29007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:15.016587 29007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:15.016671 29007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:15.020987 29007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:15.021093 29007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:15.021131 29007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:15.021194 29007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:15.021235 29007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:15.026875 29007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:15.028082 29007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:15.029213 29007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:15.037986 29012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.038331 29013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.038446 29015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:15.038751 29007 server_base.cc:1034] running on GCE node
I20260928 23:55:15.039422 29007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:15.040725 29007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:15.041932 29007 hybrid_clock.cc:648] HybridClock initialized: now 1790639715041911 us; error 33 us; skew 500 ppm
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:15.045276 29007 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:55:15.046314 29007 webserver.cc:460] Webserver started at http://127.20.88.62:35093/ using document root <none> and password file <none>
I20260928 23:55:15.046919 29007 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:15.047019 29007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:15.047271 29007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:15.049512 29007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "e7b7216ab3a14adfbf97b512781e0ea3"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "6f6462b2af9c875316fde79587866465"
server_key_iv: "9b43e6d42b41a08c15ceb86475a14bd2"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.050139 29007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "e7b7216ab3a14adfbf97b512781e0ea3"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "6f6462b2af9c875316fde79587866465"
server_key_iv: "9b43e6d42b41a08c15ceb86475a14bd2"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.054157 29007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:55:15.056727 29022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.058172 29007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:15.058353 29007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "e7b7216ab3a14adfbf97b512781e0ea3"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "6f6462b2af9c875316fde79587866465"
server_key_iv: "9b43e6d42b41a08c15ceb86475a14bd2"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.058482 29007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:15.078586 29007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:15.082134 29007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:15.082370 29007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:15.090423 29007 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42667
I20260928 23:55:15.090446 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42667 every 8 connection(s)
I20260928 23:55:15.091451 29007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:15.094496 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:15.098045 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29007
I20260928 23:55:15.098158 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:15.098459 20832 external_mini_cluster.cc:1428] Setting key 454e489885b6ad793cd7cdbfadac4e4f
I20260928 23:55:15.100651 29075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Bootstrap starting.
I20260928 23:55:15.103473 29075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:15.104387 29075 log.cc:824] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:15.106755 29075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: No bootstrap required, opened a new log
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:15.111460 29075 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:15.111759 29075 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:15.111846 29075 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b7216ab3a14adfbf97b512781e0ea3, State: Initialized, Role: FOLLOWER
I20260928 23:55:15.112402 29075 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:15.112537 29075 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:15.112622 29075 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:15.112717 29075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:15.113962 29075 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:15.114351 29075 leader_election.cc:302] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b7216ab3a14adfbf97b512781e0ea3; no voters:
I20260928 23:55:15.114688 29075 leader_election.cc:288] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:15.114805 29080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:15.115021 29080 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 LEADER]: Becoming Leader. State: Replica: e7b7216ab3a14adfbf97b512781e0ea3, State: Running, Role: LEADER
I20260928 23:55:15.115118 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.099966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.20.88.62:42667)
0928 23:55:15.100443 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:15.100462 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:15.100504 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:55:15.101315 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:15.101339 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:15.101778 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:15.102166 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:15.102897 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.104459 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.105584 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.106271 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.106439 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.109377 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:15.109386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:15.109409 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:15.109454 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:15.111504 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.112099 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.112105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.112110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.112282 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.112622 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.112625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.112627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.113129 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:15.113278 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:15.113655 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 23:55:15.114029 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":117,"threads_started":1}
I20260928 23:55:15.115350 29080 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:15.116000 29075 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:15.116722 29082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b7216ab3a14adfbf97b512781e0ea3. Latest consensus state: current_term: 1 leader_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } } }
I20260928 23:55:15.116842 29082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:15.117506 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } } }
I20260928 23:55:15.117591 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:15.117852 29087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:15.121565 29087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:15.127698 29087 catalog_manager.cc:1347] Generated new cluster ID: 5e37fe834dac42bd985170e06d3f627b
I20260928 23:55:15.127791 29087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:15.142858 29087 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:15.144167 29087 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:15.154419 29087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Generated new TSK 0
I20260928 23:55:15.155400 29087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:55:15.187525 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:15.305157 29101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:15.305511 29101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:15.305590 29101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:15.309813 29101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:15.309926 29101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:15.310024 29101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:15.315840 29101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:15.317158 29101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:15.318289 29101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:15.328469 29109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:15.328534 29101 server_base.cc:1034] running on GCE node
W20260928 23:55:15.328552 29107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.328480 29106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:15.329254 29101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:15.329993 29101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:15.331185 29101 hybrid_clock.cc:648] HybridClock initialized: now 1790639715331165 us; error 43 us; skew 500 ppm
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:15.334255 29101 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:55:15.335242 29101 webserver.cc:460] Webserver started at http://127.20.88.1:40639/ using document root <none> and password file <none>
I20260928 23:55:15.335832 29101 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:15.335897 29101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:15.336130 29101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:15.338359 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "da8abbd5fc584c7aa251281f61469133"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba353d869c21bf3ef7c8418bd75088a0"
server_key_iv: "97444b38714d6e261f5f24f1d2217805"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.338986 29101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "da8abbd5fc584c7aa251281f61469133"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba353d869c21bf3ef7c8418bd75088a0"
server_key_iv: "97444b38714d6e261f5f24f1d2217805"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.343205 29101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:15.346057 29116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.347464 29101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:55:15.347637 29101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "da8abbd5fc584c7aa251281f61469133"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba353d869c21bf3ef7c8418bd75088a0"
server_key_iv: "97444b38714d6e261f5f24f1d2217805"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.347775 29101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:15.373588 29101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:15.377009 29101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:15.377235 29101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:15.377874 29101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:15.378758 29101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:15.378834 29101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.378911 29101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:15.378955 29101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.389209 29101 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:37587
I20260928 23:55:15.389230 29229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:37587 every 8 connection(s)
I20260928 23:55:15.390204 29101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:15.395169 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29101
I20260928 23:55:15.395344 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:15.395628 20832 external_mini_cluster.cc:1428] Setting key 901f17acb60b9514dde26ba1fd7aa28a
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:55:15.406206 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.392289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:59045 (local address 127.20.88.62:42667)
0928 23:55:15.392444 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:15.392448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:15.393776 (+ 1328us) server_negotiation.cc:407] Connection header received
0928 23:55:15.394708 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:15.394711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:15.394763 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:15.394907 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:15.396522 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.397096 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.398345 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.398755 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.398881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.401934 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:15.401941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:15.401947 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:15.401987 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:15.403755 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.404359 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.404366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.404368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.404469 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.404847 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.404853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.404856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.405132 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:15.405250 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:15.405819 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 23:55:15.405985 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:55:15.407094 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.392724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:59045)
0928 23:55:15.393579 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:15.393637 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:15.394486 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:15.395082 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:15.395091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:15.395696 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:15.396356 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.396374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.397262 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.397266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.398198 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.398208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.398920 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.398923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.399086 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.399891 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:15.399909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:15.401722 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:15.403902 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.403907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.403923 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.404204 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.404569 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.404572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.404573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.404723 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.405257 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:15.405262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:15.405543 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 23:55:15.405709 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 23:55:15.406084 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":125,"threads_started":1}
I20260928 23:55:15.408591 29230 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42667
I20260928 23:55:15.408911 29230 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:55:15.409576 29230 heartbeater.cc:511] Master 127.20.88.62:42667 requested a full tablet report, sending...
I20260928 23:55:15.411486 29039 ts_manager.cc:194] Registered new tserver with Master: da8abbd5fc584c7aa251281f61469133 (127.20.88.1:37587)
I20260928 23:55:15.412779 29039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:59045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:55:15.428110 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:15.544128 29235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:15.544521 29235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:15.544569 29235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:15.548714 29235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:15.548790 29235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:15.548852 29235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:15.554639 29235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:15.555817 29235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:15.556968 29235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:15.567425 29243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.567425 29241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.567451 29240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:15.567763 29235 server_base.cc:1034] running on GCE node
I20260928 23:55:15.568212 29235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:15.568965 29235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:15.570158 29235 hybrid_clock.cc:648] HybridClock initialized: now 1790639715570137 us; error 32 us; skew 500 ppm
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:15.573123 29235 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:55:15.574065 29235 webserver.cc:460] Webserver started at http://127.20.88.2:40079/ using document root <none> and password file <none>
I20260928 23:55:15.574708 29235 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:15.574801 29235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:15.575042 29235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:15.577229 29235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "550cf77402f941d3985bc5452c3bc658"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "098aa3b771111f8f1f416c0a84500462"
server_key_iv: "789367c986ce117ebd8ae9d9581165e3"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.577808 29235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "550cf77402f941d3985bc5452c3bc658"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "098aa3b771111f8f1f416c0a84500462"
server_key_iv: "789367c986ce117ebd8ae9d9581165e3"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.581735 29235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:15.584498 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.585945 29235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:55:15.586088 29235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "550cf77402f941d3985bc5452c3bc658"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "098aa3b771111f8f1f416c0a84500462"
server_key_iv: "789367c986ce117ebd8ae9d9581165e3"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.586225 29235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:15.614583 29235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:15.618628 29235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:15.618867 29235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:15.619534 29235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:15.620509 29235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:15.620565 29235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.620633 29235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:15.620671 29235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.630718 29235 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:33433
I20260928 23:55:15.630744 29363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:33433 every 8 connection(s)
I20260928 23:55:15.631796 29235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:15.635540 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29235
I20260928 23:55:15.635684 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:15.635977 20832 external_mini_cluster.cc:1428] Setting key 23a0899d5b3b35a5356b4620ae7a2e48
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:55:15.648571 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.633946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:60273 (local address 127.20.88.62:42667)
0928 23:55:15.634111 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:15.634117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:15.635313 (+ 1196us) server_negotiation.cc:407] Connection header received
0928 23:55:15.636521 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:15.636525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:15.636611 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:15.636698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:15.638415 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.638948 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.640374 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.640784 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.640919 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.643969 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:15.643975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:15.643979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:15.644018 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:15.645878 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.646526 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.646530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.646531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.646619 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.647052 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.647056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.647057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.647400 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:15.647497 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:15.648102 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 23:55:15.648320 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:55:15.649472 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.634416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:60273)
0928 23:55:15.635133 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:15.635176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:15.636205 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:15.636916 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:15.636924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:15.637442 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:15.638219 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.638258 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.639094 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.639100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.640161 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.640174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.640970 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.640973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.641218 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.642003 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:15.642021 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:15.643739 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:15.646021 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.646027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.646045 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.646399 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.646714 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.646717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.646718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.646891 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.647515 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:15.647521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:15.647800 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 23:55:15.647995 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 23:55:15.648381 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":138,"threads_started":1}
I20260928 23:55:15.650810 29364 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42667
I20260928 23:55:15.651111 29364 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:15.651769 29364 heartbeater.cc:511] Master 127.20.88.62:42667 requested a full tablet report, sending...
I20260928 23:55:15.653069 29039 ts_manager.cc:194] Registered new tserver with Master: 550cf77402f941d3985bc5452c3bc658 (127.20.88.2:33433)
I20260928 23:55:15.654057 29039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:60273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:55:15.667902 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:15.784610 29369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:15.784912 29369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:15.784958 29369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:15.789060 29369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:15.789132 29369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:15.789196 29369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:15.795123 29369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:15.796312 29369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:15.797362 29369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:15.807607 29374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.807607 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:15.807601 29375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:15.807911 29369 server_base.cc:1034] running on GCE node
I20260928 23:55:15.808503 29369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:15.809228 29369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:15.810431 29369 hybrid_clock.cc:648] HybridClock initialized: now 1790639715810412 us; error 35 us; skew 500 ppm
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:15.813567 29369 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:55:15.814554 29369 webserver.cc:460] Webserver started at http://127.20.88.3:37745/ using document root <none> and password file <none>
I20260928 23:55:15.815162 29369 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:15.815227 29369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:15.815415 29369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:15.818037 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "0f25bd5d2dae4dbd834d2c7e720fc60e"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba5755fbacca78f10c87f223cd153315"
server_key_iv: "9a4cca6b9bec97c05b89c6ec5036cd83"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.818729 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "0f25bd5d2dae4dbd834d2c7e720fc60e"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba5755fbacca78f10c87f223cd153315"
server_key_iv: "9a4cca6b9bec97c05b89c6ec5036cd83"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.823201 29369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:15.826251 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.827773 29369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:15.827909 29369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "0f25bd5d2dae4dbd834d2c7e720fc60e"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba5755fbacca78f10c87f223cd153315"
server_key_iv: "9a4cca6b9bec97c05b89c6ec5036cd83"
server_key_version: "encryptionkey@0"
I20260928 23:55:15.828027 29369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:15.841156 29369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:15.844818 29369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:15.845009 29369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:15.845615 29369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:15.846515 29369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:15.846570 29369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.846616 29369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:15.846632 29369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:15.857319 29369 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:33317
I20260928 23:55:15.857340 29497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:33317 every 8 connection(s)
I20260928 23:55:15.858381 29369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:15.865206 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29369
I20260928 23:55:15.865360 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:15.865696 20832 external_mini_cluster.cc:1428] Setting key 907d7fd186e052db26add809e73f193f
Sep 28 23:55:15 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:15.873512 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.860736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53763 (local address 127.20.88.62:42667)
0928 23:55:15.860948 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:15.860952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:15.862136 (+ 1184us) server_negotiation.cc:407] Connection header received
0928 23:55:15.863084 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:15.863087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:15.863137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:15.863241 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:15.864593 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.865305 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.866791 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.867099 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.867194 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.869901 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:15.869905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:15.869907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:15.869933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:15.871355 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.871874 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.871877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.871878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.871958 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:15.872398 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:15.872401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:15.872402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:15.872593 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:15.872671 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:15.873240 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 23:55:15.873387 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 23:55:15.874313 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:15.861148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:53763)
0928 23:55:15.861869 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:15.862007 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:15.862845 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:15.863455 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:15.863462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:15.863840 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:15.864452 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.864468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.865481 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.865484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.866657 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:15.866667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:15.867283 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:15.867287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:15.867491 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:15.868197 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:15.868214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:15.869729 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:15.871470 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.871475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.871490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.871768 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.872072 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:15.872075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:15.872076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:15.872293 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:15.872687 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:15.872692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:15.872964 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 23:55:15.873129 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 23:55:15.873411 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":133,"threads_started":1}
I20260928 23:55:15.875607 29498 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42667
I20260928 23:55:15.875902 29498 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:15.876506 29498 heartbeater.cc:511] Master 127.20.88.62:42667 requested a full tablet report, sending...
I20260928 23:55:15.877626 29039 ts_manager.cc:194] Registered new tserver with Master: 0f25bd5d2dae4dbd834d2c7e720fc60e (127.20.88.3:33317)
I20260928 23:55:15.878234 29039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:53763
I20260928 23:55:15.879839 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
Entry for principal oryx/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab.
WARNING: no policy specified for oryx/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
Entry for principal oryx/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab.
WARNING: no policy specified for oryx/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
Entry for principal oryx/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab.
I20260928 23:55:15.963850 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29101
I20260928 23:55:15.969703 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29235
I20260928 23:55:15.975266 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29369
I20260928 23:55:15.980870 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29007
I20260928 23:55:15.987452 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42667
--webserver_interface=127.20.88.62
--webserver_port=35093
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:16.105239 29507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:16.105602 29507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:16.105679 29507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:16.109946 29507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:16.110095 29507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:16.110134 29507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:16.110176 29507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:16.110206 29507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:16.115993 29507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:16.117322 29507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:16.118602 29507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:16.130640 29515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.130671 29513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:16.130729 29507 server_base.cc:1034] running on GCE node
W20260928 23:55:16.130645 29512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:16.131702 29507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:16.132979 29507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:16.134158 29507 hybrid_clock.cc:648] HybridClock initialized: now 1790639716134133 us; error 40 us; skew 500 ppm
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639716, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:16.137116 29507 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:55:16.138020 29507 webserver.cc:460] Webserver started at http://127.20.88.62:35093/ using document root <none> and password file <none>
I20260928 23:55:16.138584 29507 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:16.138664 29507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:16.142695 29507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:55:16.144984 29522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.146329 29507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:16.146533 29507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "e7b7216ab3a14adfbf97b512781e0ea3"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "6f6462b2af9c875316fde79587866465"
server_key_iv: "9b43e6d42b41a08c15ceb86475a14bd2"
server_key_version: "encryptionkey@0"
I20260928 23:55:16.147123 29507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:16.169952 29507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:16.173322 29507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:16.173558 29507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:16.181334 29507 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42667
I20260928 23:55:16.181353 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42667 every 8 connection(s)
I20260928 23:55:16.182523 29507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:16.184831 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29507
I20260928 23:55:16.185284 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:37587
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:42667
--webserver_port=40639
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:55:16.187554 29575 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:55:16.189472 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Bootstrap starting.
I20260928 23:55:16.206655 29575 log.cc:824] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:16.213896 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:55:16.214419 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Bootstrap complete.
I20260928 23:55:16.218894 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:16.219339 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b7216ab3a14adfbf97b512781e0ea3, State: Initialized, Role: FOLLOWER
I20260928 23:55:16.219913 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:16.220047 29575 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:16.220148 29575 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:16.220295 29575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:55:16.222402 29575 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:16.222781 29575 leader_election.cc:302] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b7216ab3a14adfbf97b512781e0ea3; no voters:
I20260928 23:55:16.223165 29575 leader_election.cc:288] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:55:16.223269 29579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:55:16.223479 29579 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [term 2 LEADER]: Becoming Leader. State: Replica: e7b7216ab3a14adfbf97b512781e0ea3, State: Running, Role: LEADER
I20260928 23:55:16.223789 29579 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } }
I20260928 23:55:16.224538 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:16.225205 29580 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } } }
I20260928 23:55:16.225332 29580 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:16.226028 29581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b7216ab3a14adfbf97b512781e0ea3. Latest consensus state: current_term: 2 leader_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b7216ab3a14adfbf97b512781e0ea3" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42667 } } }
I20260928 23:55:16.226181 29581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:16.226413 29586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:16.230029 29586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:16.230702 29586 catalog_manager.cc:1259] Loaded cluster ID: 5e37fe834dac42bd985170e06d3f627b
I20260928 23:55:16.230762 29586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:16.233342 29586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:16.234114 29586 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7b7216ab3a14adfbf97b512781e0ea3: Loaded TSK: 0
I20260928 23:55:16.235369 29586 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:55:16.308712 29577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:16.309047 29577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:16.309096 29577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:16.313640 29577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:16.313722 29577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:16.313788 29577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:16.319568 29577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.1.keytab
--principal=oryx/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:37587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=40639
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:16.320792 29577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:16.321987 29577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:16.332751 29601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.332801 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.332756 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:16.333424 29577 server_base.cc:1034] running on GCE node
I20260928 23:55:16.333987 29577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:16.334887 29577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:16.336110 29577 hybrid_clock.cc:648] HybridClock initialized: now 1790639716336082 us; error 36 us; skew 500 ppm
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639716, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:16.339929 29577 init.cc:377] Logged in from keytab as oryx/127.20.88.1@KRBTEST.COM (short username oryx)
I20260928 23:55:16.341109 29577 webserver.cc:460] Webserver started at http://127.20.88.1:40639/ using document root <none> and password file <none>
I20260928 23:55:16.341741 29577 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:16.341861 29577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:16.346350 29577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:55:16.349151 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.350852 29577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:16.351058 29577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "da8abbd5fc584c7aa251281f61469133"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba353d869c21bf3ef7c8418bd75088a0"
server_key_iv: "97444b38714d6e261f5f24f1d2217805"
server_key_version: "encryptionkey@0"
I20260928 23:55:16.351774 29577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:16.369463 29577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:16.373133 29577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:16.373368 29577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:16.374008 29577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:16.374946 29577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:16.375020 29577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.375097 29577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:16.375141 29577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.385778 29577 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:37587
I20260928 23:55:16.385885 29724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:37587 every 8 connection(s)
I20260928 23:55:16.387262 29577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:16.393007 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29577
I20260928 23:55:16.393563 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:33433
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:42667
--webserver_port=40079
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.403512 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.390051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:49007)
0928 23:55:16.390792 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.390848 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.391979 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.393336 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.393349 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.393900 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.394671 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.394698 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.396950 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.396954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.398000 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.398013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.399278 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.399282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.399433 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.400110 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.400128 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.402678 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":139,"threads_started":1}
W20260928 23:55:16.404084 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.389902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:49007 (local address 127.20.88.62:42667)
0928 23:55:16.390447 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.390464 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.391036 (+ 572us) server_negotiation.cc:407] Connection header received
0928 23:55:16.392527 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.392539 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.392873 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.393225 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.394906 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.396761 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.398156 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.399068 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.399291 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.403049 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:49007: BlockingRecv error: recv got EOF from 127.20.88.1:49007 (error 108)
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
W20260928 23:55:16.404194 29725 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42667 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.88.62:42667: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.410000 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.404817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:37715)
0928 23:55:16.404920 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.404938 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.405030 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.405274 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.405277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.405451 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.405544 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.405550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.406304 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.406309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.406861 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.406869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.407332 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.407335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.407441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.407957 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.407975 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.409854 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:55:16.410439 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.404918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:37715 (local address 127.20.88.62:42667)
0928 23:55:16.405048 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.405054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.405071 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:55:16.405116 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.405119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.405171 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.405260 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.405659 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.406194 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.406989 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.407219 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.407328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.410008 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:37715: BlockingRecv error: recv got EOF from 127.20.88.1:37715 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.416224 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.410580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:53307)
0928 23:55:16.410700 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.410716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.410794 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.411173 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.411175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.411300 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.411375 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.411379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.412415 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.412419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.412949 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.412957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.413526 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.413529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.413590 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.414164 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.414178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.416086 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 23:55:16.416448 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.410813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:53307 (local address 127.20.88.62:42667)
0928 23:55:16.410944 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.410949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.410963 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:16.411006 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.411009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.411054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.411147 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.411478 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.412309 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.413060 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.413404 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.413521 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.416291 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:53307: BlockingRecv error: recv got EOF from 127.20.88.1:53307 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 23:55:16.416590 29725 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:55:16.518172 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:16.518473 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:16.518519 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:16.522823 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:16.522902 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:16.523001 29729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:16.529057 29729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.2.keytab
--principal=oryx/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:33433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=40079
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:16.530176 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:16.531226 29729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:16.541630 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.541651 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.541630 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:16.541916 29729 server_base.cc:1034] running on GCE node
I20260928 23:55:16.542532 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:16.543308 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:16.544504 29729 hybrid_clock.cc:648] HybridClock initialized: now 1790639716544481 us; error 40 us; skew 500 ppm
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639716, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:16.547531 29729 init.cc:377] Logged in from keytab as oryx/127.20.88.2@KRBTEST.COM (short username oryx)
I20260928 23:55:16.548623 29729 webserver.cc:460] Webserver started at http://127.20.88.2:40079/ using document root <none> and password file <none>
I20260928 23:55:16.549248 29729 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:16.549384 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:16.553867 29729 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:55:16.556349 29744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.557713 29729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:16.557860 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "550cf77402f941d3985bc5452c3bc658"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "098aa3b771111f8f1f416c0a84500462"
server_key_iv: "789367c986ce117ebd8ae9d9581165e3"
server_key_version: "encryptionkey@0"
I20260928 23:55:16.558458 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:16.569628 29729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:16.573104 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:16.573339 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:16.573978 29729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:16.574853 29729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:16.574925 29729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.575004 29729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:16.575047 29729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.584859 29729 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:33433
I20260928 23:55:16.584883 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:33433 every 8 connection(s)
I20260928 23:55:16.585971 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:16.591619 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29729
I20260928 23:55:16.592136 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:33317
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:42667
--webserver_port=37745
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:55:16.598446 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.588107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:40279 (local address 127.20.88.62:42667)
0928 23:55:16.588342 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.588345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.589552 (+ 1207us) server_negotiation.cc:407] Connection header received
0928 23:55:16.590531 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.590535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.590603 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.590710 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.592319 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.592906 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.594343 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.594576 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.594698 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.598312 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:40279: BlockingRecv error: recv got EOF from 127.20.88.2:40279 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 23:55:16.599190 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.588667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:40279)
0928 23:55:16.589372 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.589425 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.590272 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.590907 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.590916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.591326 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.592073 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.592098 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.593053 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.593060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.594200 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.594210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.594745 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.594748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.594933 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.595629 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.595652 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.598171 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":128,"threads_started":1}
W20260928 23:55:16.599691 29858 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42667 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.88.62:42667: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.605453 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.600143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:48675)
0928 23:55:16.600285 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.600302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.600418 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.600705 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.600707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.600878 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.600974 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.600979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.601730 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.601735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.602281 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.602289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.602744 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.602747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.602841 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.603361 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.603373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.605309 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:55:16.605625 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.600334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:48675 (local address 127.20.88.62:42667)
0928 23:55:16.600497 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.600502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.600517 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:16.600562 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.600564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.600601 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.600690 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.601097 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.601617 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.602405 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.602630 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.602742 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.605460 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:48675: BlockingRecv error: recv got EOF from 127.20.88.2:48675 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.612026 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.606058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:42845)
0928 23:55:16.606388 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.606403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.606488 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.606730 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.606733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.606905 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.606998 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.607004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.607737 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.607743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.608724 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.608735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.609218 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.609221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.609323 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.609816 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.609830 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.611867 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
W20260928 23:55:16.612146 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.606155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:42845 (local address 127.20.88.62:42667)
0928 23:55:16.606273 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.606277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.606464 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:55:16.606576 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.606578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.606618 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.606684 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.607099 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.607619 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.608870 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.609094 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.609188 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.612034 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:42845: BlockingRecv error: recv got EOF from 127.20.88.2:42845 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:55:16.612506 29858 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:55:16.714769 29861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:16.715147 29861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:16.715230 29861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:16.719620 29861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:16.719755 29861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:16.719869 29861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:16.725860 29861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/krb5kdc/oryx_127.20.88.3.keytab
--principal=oryx/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:33317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=37745
--tserver_master_addrs=127.20.88.62:42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:16.727506 29861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:16.728648 29861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:16.739274 29867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.739293 29868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:16.739382 29870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:16.739573 29861 server_base.cc:1034] running on GCE node
I20260928 23:55:16.740015 29861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:16.740749 29861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:16.741930 29861 hybrid_clock.cc:648] HybridClock initialized: now 1790639716741905 us; error 36 us; skew 500 ppm
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639716, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:16.744983 29861 init.cc:377] Logged in from keytab as oryx/127.20.88.3@KRBTEST.COM (short username oryx)
I20260928 23:55:16.745983 29861 webserver.cc:460] Webserver started at http://127.20.88.3:37745/ using document root <none> and password file <none>
I20260928 23:55:16.746588 29861 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:16.746690 29861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:16.751063 29861 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:55:16.753440 29877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.754860 29861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:16.755050 29861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "0f25bd5d2dae4dbd834d2c7e720fc60e"
format_stamp: "Formatted at 2026-09-28 23:55:15 on dist-test-slave-wvpw"
server_key: "ba5755fbacca78f10c87f223cd153315"
server_key_iv: "9a4cca6b9bec97c05b89c6ec5036cd83"
server_key_version: "encryptionkey@0"
I20260928 23:55:16.755864 29861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:16.767534 29861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:16.770850 29861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:16.771068 29861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:16.771693 29861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:16.772586 29861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:16.772634 29861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.772673 29861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:16.772730 29861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:16.781996 29861 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:33317
I20260928 23:55:16.782074 29990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:33317 every 8 connection(s)
I20260928 23:55:16.783070 29861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:16.789309 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 29861
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:55:16.795532 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.785047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44199 (local address 127.20.88.62:42667)
0928 23:55:16.785215 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.785219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.786301 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 23:55:16.787200 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.787203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.787253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.787339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.788658 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.789512 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.791014 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.791225 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.791299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.795406 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:44199: BlockingRecv error: recv got EOF from 127.20.88.3:44199 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 23:55:16.796408 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.785442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:44199)
0928 23:55:16.786146 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.786189 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.786975 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.787547 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.787554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.787935 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.788478 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.788507 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.789670 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.789673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.790882 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.790895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.791399 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.791405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.791635 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.792429 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.792447 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.795290 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":114,"threads_started":1}
W20260928 23:55:16.796990 29991 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42667 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.88.62:42667: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
I20260928 23:55:16.800583 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.790448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.20.88.62:42667)
0928 23:55:16.790682 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.790685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.790695 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:55:16.790727 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.790729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.790766 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.790840 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.791824 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.792540 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.793663 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.793907 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.794065 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.795641 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:16.795650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:16.795674 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:16.795710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:16.798069 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:16.798610 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:16.798616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:16.798623 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:16.798788 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:16.799138 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:16.799142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:16.799144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:16.799624 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:16.799781 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:16.800119 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 23:55:16.800456 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.803491 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.797506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:38487)
0928 23:55:16.797607 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.797622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.797706 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.798025 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.798028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.798222 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.798370 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.798380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.799456 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.799459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.800293 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.800305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.800808 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.800812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.800913 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.801429 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.801444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.803393 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:55:16.803671 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.797587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:38487 (local address 127.20.88.62:42667)
0928 23:55:16.797714 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.797719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.797761 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:55:16.797804 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.797807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.797854 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.797935 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.798574 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.799331 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.800419 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.800688 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.800805 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.803556 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:38487: BlockingRecv error: recv got EOF from 127.20.88.3:38487 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:16 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:16.809024 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:16.804081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:51729)
0928 23:55:16.804219 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:16.804261 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:16.804346 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:16.804580 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:16.804583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:16.804749 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:16.804826 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.804837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.805581 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.805586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.806125 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:16.806133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.806595 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.806599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:16.806679 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.807114 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:16.807125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:16.808921 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 23:55:16.809177 29727 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:16.804154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:51729 (local address 127.20.88.62:42667)
0928 23:55:16.804321 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:16.804325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:16.804347 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:55:16.804428 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:16.804431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:16.804472 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:16.804565 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:16.804946 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.805467 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.806246 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:16.806483 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:16.806593 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:16.809053 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:51729: BlockingRecv error: recv got EOF from 127.20.88.3:51729 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 23:55:16.809314 29991 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:17.423408 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:17.417390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:52049)
0928 23:55:17.417762 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:17.417782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:17.417928 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:17.418371 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:17.418374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:17.418598 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:17.418699 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.418705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.419707 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.419710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.420437 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.420448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.420879 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.420883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.420949 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.421438 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:17.421450 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:17.423265 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260928 23:55:17.423620 29997 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:17.417616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:52049 (local address 127.20.88.62:42667)
0928 23:55:17.417925 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:17.417930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:17.417953 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:55:17.418096 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:17.418123 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:17.418184 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:17.418286 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:17.418809 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.419540 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.420544 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.420766 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.420844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.423427 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:52049: BlockingRecv error: recv got EOF from 127.20.88.1:52049 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:17.618947 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:17.613207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:49161)
0928 23:55:17.613521 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:17.613534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:17.613636 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:17.613843 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:17.613845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:17.614023 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:17.614138 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.614142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.614935 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.614938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.615590 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.615601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.616059 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.616061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.616137 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.616816 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:17.616828 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:17.618791 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":138,"threads_started":1}
W20260928 23:55:17.619169 29997 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:17.613250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:49161 (local address 127.20.88.62:42667)
0928 23:55:17.613438 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:17.613441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:17.613639 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:55:17.613704 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:17.613706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:17.613757 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:17.613830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:17.614268 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.614800 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.615717 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.615946 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.616034 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.618975 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:49161: BlockingRecv error: recv got EOF from 127.20.88.2:49161 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:17 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:17.815828 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:17.809985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:57177)
0928 23:55:17.810333 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:17.810348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:17.810458 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:17.810685 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:17.810687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:17.810869 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:17.810967 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.810973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.811777 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.811779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.812413 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:17.812422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.812842 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.812845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:17.812911 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.813544 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:17.813556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:17.815684 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
W20260928 23:55:17.816001 29997 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:17.810095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:57177 (local address 127.20.88.62:42667)
0928 23:55:17.810341 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:17.810345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:17.810407 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:55:17.810528 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:17.810530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:17.810576 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:17.810644 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:17.811088 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.811670 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.812512 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:17.812733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:17.812821 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:17.815866 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:57177: BlockingRecv error: recv got EOF from 127.20.88.3:57177 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:18.430112 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:18.424148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:33667)
0928 23:55:18.424456 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:18.424472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:18.424565 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:18.424824 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:18.424827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:18.425101 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:18.425199 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.425205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.426139 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.426142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.426703 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.426711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.427226 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.427230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.427336 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.427869 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:18.427881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:18.429969 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
W20260928 23:55:18.430357 30001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:18.424309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:33667 (local address 127.20.88.62:42667)
0928 23:55:18.424554 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:18.424557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:18.424570 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:55:18.424639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:18.424641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:18.424681 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:18.424755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:18.425349 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.425995 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.426858 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.427101 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.427218 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.430188 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:33667: BlockingRecv error: recv got EOF from 127.20.88.1:33667 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:18.625756 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:18.619810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:52527)
0928 23:55:18.620162 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:18.620178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:18.620382 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:18.620680 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:18.620683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:18.620903 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:18.620996 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.621002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.621773 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.621779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.622353 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.622362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.622839 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.622843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.622928 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.623493 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:18.623506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:18.625623 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":87,"threads_started":1}
W20260928 23:55:18.625950 30001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:18.619975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:52527 (local address 127.20.88.62:42667)
0928 23:55:18.620138 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:18.620143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:18.620302 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:55:18.620509 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:18.620512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:18.620569 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:18.620661 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:18.621118 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.621656 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.622475 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.622711 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.622823 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.625782 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:52527: BlockingRecv error: recv got EOF from 127.20.88.2:52527 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:18 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:18.831059 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:18.817013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:37033)
0928 23:55:18.817348 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:18.817364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:18.817519 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:18.817835 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:18.817838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:18.818084 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:18.818192 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.818200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.819072 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.819079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.819659 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:18.819669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.820210 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.820215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:18.820343 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.828804 (+ 8461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:18.828821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:18.830895 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
W20260928 23:55:18.831238 30001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:18.817344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:37033 (local address 127.20.88.62:42667)
0928 23:55:18.817500 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:18.817507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:18.817525 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:55:18.817627 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:18.817631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:18.817693 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:18.817807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:18.818354 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.818924 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.819835 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:18.820071 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:18.820210 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:18.831098 (+ 10888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:37033: BlockingRecv error: recv got EOF from 127.20.88.3:37033 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:19.436775 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:19.430985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:38903)
0928 23:55:19.431390 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:19.431412 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:19.431516 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:19.431803 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:19.431806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:19.432005 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:19.432104 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.432111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.432989 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.432991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.433550 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.433557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.433984 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.433986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.434045 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.434564 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:19.434576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:19.436578 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
W20260928 23:55:19.436869 30005 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:19.431124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38903 (local address 127.20.88.62:42667)
0928 23:55:19.431375 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:19.431378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:19.431487 (+ 109us) server_negotiation.cc:407] Connection header received
0928 23:55:19.431603 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:19.431605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:19.431649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:19.431731 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:19.432316 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.432876 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.433656 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.433884 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.433963 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.436732 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:38903: BlockingRecv error: recv got EOF from 127.20.88.1:38903 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":59,"threads_started":1}
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:19.632778 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:19.626600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:45539)
0928 23:55:19.626928 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:19.626945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:19.627106 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:19.627445 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:19.627448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:19.627671 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:19.627787 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.627794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.628775 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.628780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.629366 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.629374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.629911 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.629915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.630009 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.630530 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:19.630551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:19.632584 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
W20260928 23:55:19.632851 30005 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:19.626659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:45539 (local address 127.20.88.62:42667)
0928 23:55:19.626906 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:19.626911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:19.627040 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:55:19.627226 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:19.627229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:19.627322 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:19.627422 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:19.627923 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.628659 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.629497 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.629782 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.629882 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.632736 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:45539: BlockingRecv error: recv got EOF from 127.20.88.2:45539 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:19 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:19.837878 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:19.832001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:44881)
0928 23:55:19.832432 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:19.832446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:19.832552 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:19.832798 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:19.832800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:19.832988 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:19.833090 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.833096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.833908 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.833911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.834505 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:19.834514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.835031 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.835034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:19.835116 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.835700 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:19.835715 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:19.837718 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":106,"threads_started":1}
W20260928 23:55:19.838047 30005 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:19.832072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44881 (local address 127.20.88.62:42667)
0928 23:55:19.832258 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:19.832263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:19.832552 (+ 289us) server_negotiation.cc:407] Connection header received
0928 23:55:19.832621 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:19.832623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:19.832675 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:19.832759 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:19.833205 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.833777 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.834635 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:19.834893 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:19.834989 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:19.837909 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:44881: BlockingRecv error: recv got EOF from 127.20.88.3:44881 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:20.442987 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:20.437613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:45207)
0928 23:55:20.437895 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:20.437908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:20.437999 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:20.438238 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:20.438240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:20.438428 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:20.438528 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.438533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.439279 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.439282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.439858 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.439867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.440370 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.440373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.440435 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.440936 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:20.440947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:20.442865 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
W20260928 23:55:20.443234 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:20.437737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:45207 (local address 127.20.88.62:42667)
0928 23:55:20.437995 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:20.437999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:20.438013 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:20.438080 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:20.438082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:20.438127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:20.438201 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:20.438633 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.439167 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.439992 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.440224 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.440360 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.443050 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:45207: BlockingRecv error: recv got EOF from 127.20.88.1:45207 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:20.639649 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:20.633691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:41503)
0928 23:55:20.633972 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:20.633985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:20.634076 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:20.634413 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:20.634415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:20.634599 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:20.634700 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.634706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.635606 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.635610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.636207 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.636216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.636861 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.636864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.636949 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.637518 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:20.637530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:20.639511 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20260928 23:55:20.639832 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:20.633870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:41503 (local address 127.20.88.62:42667)
0928 23:55:20.634026 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:20.634029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:20.634044 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:20.634211 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:20.634214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:20.634275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:20.634356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:20.634841 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.635467 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.636433 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.636680 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.636807 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.639688 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:41503: BlockingRecv error: recv got EOF from 127.20.88.2:41503 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:20 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:20.845193 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:20.838942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:47057)
0928 23:55:20.839283 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:20.839297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:20.839446 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:20.839768 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:20.839771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:20.839983 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:20.840079 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.840085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.840934 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.840939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.841542 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:20.841551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.842053 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.842057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:20.842147 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.842736 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:20.842750 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:20.845018 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
W20260928 23:55:20.845606 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:20.839038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:47057 (local address 127.20.88.62:42667)
0928 23:55:20.839233 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:20.839238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:20.839384 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:55:20.839560 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:20.839564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:20.839623 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:20.839749 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:20.840212 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.840809 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.841670 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:20.841921 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:20.842044 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:20.845176 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:47057: BlockingRecv error: recv got EOF from 127.20.88.3:47057 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:21.450009 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:21.444104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:43427)
0928 23:55:21.444441 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:21.444458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:21.444567 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:21.444819 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:21.444821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:21.445027 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:21.445121 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.445127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.445918 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.445921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.446505 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.446514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.447028 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.447031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.447093 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.447727 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:21.447740 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:21.449865 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
W20260928 23:55:21.450250 30013 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:21.444202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:43427 (local address 127.20.88.62:42667)
0928 23:55:21.444544 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:21.444548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:21.444563 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:21.444656 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:21.444658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:21.444704 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:21.444784 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:21.445237 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.445778 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.446643 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.446913 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.447001 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.450098 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:43427: BlockingRecv error: recv got EOF from 127.20.88.1:43427 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":82,"threads_started":1}
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:21.646929 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:21.640741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:54757)
0928 23:55:21.641066 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:21.641080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:21.641173 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:21.641536 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:21.641539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:21.641739 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:21.641839 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.641845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.642713 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.642718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.643330 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.643340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.643830 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.643833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.643907 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.644570 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:21.644588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:21.646750 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":135,"threads_started":1}
W20260928 23:55:21.647065 30013 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:21.640955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:54757 (local address 127.20.88.62:42667)
0928 23:55:21.641102 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:21.641106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:21.641124 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:55:21.641342 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:21.641344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:21.641394 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:21.641486 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:21.641984 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.642591 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.643460 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.643691 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.643777 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.646910 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:54757: BlockingRecv error: recv got EOF from 127.20.88.2:54757 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:21 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:21.852180 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:21.846258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:56677)
0928 23:55:21.846666 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:21.846684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:21.846843 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:21.847175 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:21.847178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:21.847373 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:21.847471 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.847476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.848224 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.848253 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.848822 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:21.848830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.849268 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.849271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:21.849341 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.849879 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:21.849891 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:21.852040 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":112,"threads_started":1}
W20260928 23:55:21.852421 30013 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:21.846475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:56677 (local address 127.20.88.62:42667)
0928 23:55:21.846666 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:21.846672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:21.846783 (+ 111us) server_negotiation.cc:407] Connection header received
0928 23:55:21.846953 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:21.846956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:21.847037 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:21.847127 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:21.847589 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.848124 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.848933 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:21.849157 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:21.849257 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:21.852298 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:56677: BlockingRecv error: recv got EOF from 127.20.88.3:56677 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:22.457017 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:22.451057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:44745)
0928 23:55:22.451318 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:22.451335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:22.451442 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:22.451746 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:22.451749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:22.452031 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:22.452134 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.452141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.452943 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.452946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.453577 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.453585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.454052 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.454054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.454121 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.454639 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:22.454652 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:22.456867 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
W20260928 23:55:22.457319 30016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:22.450974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:44745 (local address 127.20.88.62:42667)
0928 23:55:22.451319 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:22.451323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:22.451403 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:55:22.451560 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:22.451563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:22.451609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:22.451691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:22.452287 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.452840 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.453697 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.453941 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.454026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.457030 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:44745: BlockingRecv error: recv got EOF from 127.20.88.1:44745 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:22.653509 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:22.647790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:43343)
0928 23:55:22.648084 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:22.648097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:22.648192 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:22.648545 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:22.648550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:22.648774 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:22.648917 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.648925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.649790 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.649792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.650371 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.650379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.650878 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.650880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.650941 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.651523 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:22.651536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:22.653385 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
W20260928 23:55:22.653690 30016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:22.647928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:43343 (local address 127.20.88.62:42667)
0928 23:55:22.648073 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:22.648077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:22.648146 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:55:22.648337 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:22.648339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:22.648389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:22.648463 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:22.649068 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.649671 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.650492 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.650735 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.650880 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.653549 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:43343: BlockingRecv error: recv got EOF from 127.20.88.2:43343 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:22 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:22.858637 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:22.853051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:57513)
0928 23:55:22.853331 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:22.853345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:22.853447 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:22.853679 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:22.853682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:22.853859 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:22.853951 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.853956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.854718 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.854720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.855360 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:22.855370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.855787 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.855790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:22.855850 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.856510 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:22.856521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:22.858517 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
W20260928 23:55:22.858884 30016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:22.853155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:57513 (local address 127.20.88.62:42667)
0928 23:55:22.853348 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:22.853352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:22.853414 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:55:22.853526 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:22.853528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:22.853574 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:22.853643 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:22.854061 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.854621 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.855471 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:22.855690 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:22.855780 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:22.858750 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:57513: BlockingRecv error: recv got EOF from 127.20.88.3:57513 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:23.463932 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:23.457850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:33999)
0928 23:55:23.458189 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:23.458203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:23.458297 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:23.458562 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:23.458564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:23.458756 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:23.458858 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.458863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.459765 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.459768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.460452 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.460463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.460991 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.460993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.461058 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.461665 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:23.461677 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:23.463745 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
W20260928 23:55:23.464056 30021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:23.457924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:33999 (local address 127.20.88.62:42667)
0928 23:55:23.458228 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:23.458232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:23.458251 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:55:23.458394 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:23.458396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:23.458443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:23.458531 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:23.458985 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.459646 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.460587 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.460865 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.460969 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.463900 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:33999: BlockingRecv error: recv got EOF from 127.20.88.1:33999 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:23.660408 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:23.654354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:47481)
0928 23:55:23.654743 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:23.654762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:23.654921 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:23.655203 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:23.655206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:23.655467 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:23.655573 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.655579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.656500 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.656527 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.657148 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.657159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.657594 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.657597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.657660 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.658160 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:23.658172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:23.660204 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":113,"threads_started":1}
W20260928 23:55:23.660475 30021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:23.654529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:47481 (local address 127.20.88.62:42667)
0928 23:55:23.654719 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:23.654724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:23.654851 (+ 127us) server_negotiation.cc:407] Connection header received
0928 23:55:23.655027 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:23.655030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:23.655086 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:23.655181 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:23.655699 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.656358 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.657278 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.657494 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.657571 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.660363 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:47481: BlockingRecv error: recv got EOF from 127.20.88.2:47481 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:23 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:23.871605 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:23.859519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:57033)
0928 23:55:23.859905 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:23.859919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:23.860067 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:23.860500 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:23.860502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:23.860686 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:23.860784 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.860790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.861698 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.861701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.862329 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:23.862339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.862949 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.862953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:23.863021 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.869193 (+ 6172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:23.869208 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:23.871478 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
W20260928 23:55:23.871836 30021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:23.859587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:57033 (local address 127.20.88.62:42667)
0928 23:55:23.859761 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:23.859767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:23.860002 (+ 235us) server_negotiation.cc:407] Connection header received
0928 23:55:23.860335 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:23.860338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:23.860393 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:23.860466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:23.860956 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.861581 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.862446 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:23.862718 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:23.862812 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:23.871687 (+ 8875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:57033: BlockingRecv error: recv got EOF from 127.20.88.3:57033 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:24.471276 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:24.465020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:40737)
0928 23:55:24.465314 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:24.465328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:24.465431 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:24.465784 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:24.465803 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:24.465988 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:24.466090 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.466096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.466894 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.466897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.467481 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.467490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.468011 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.468015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.468097 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.468802 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:24.468825 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:24.471091 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
W20260928 23:55:24.471377 30025 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:24.465231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:40737 (local address 127.20.88.62:42667)
0928 23:55:24.465515 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:24.465519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:24.465533 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:24.465577 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:24.465579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:24.465623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:24.465708 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:24.466212 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.466769 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.467603 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.467850 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.467932 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.471236 (+ 3304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:40737: BlockingRecv error: recv got EOF from 127.20.88.1:40737 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:24.672322 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:24.661425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:44259)
0928 23:55:24.661739 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:24.661754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:24.661850 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:24.662115 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:24.662117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:24.662305 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:24.662400 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.662405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.663564 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.663568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.664364 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.664375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.664836 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.664839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.664907 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.669877 (+ 4970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:24.669892 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:24.672083 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":122,"threads_started":1}
W20260928 23:55:24.672432 30025 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:24.661682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:44259 (local address 127.20.88.62:42667)
0928 23:55:24.661824 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:24.661828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:24.661844 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:24.661939 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:24.661941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:24.661988 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:24.662074 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:24.662519 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.663448 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.664482 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.664707 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.664807 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.672302 (+ 7495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:44259: BlockingRecv error: recv got EOF from 127.20.88.2:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:24 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:24.878309 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:24.872435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:43389)
0928 23:55:24.872790 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:24.872807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:24.872968 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:24.873259 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:24.873262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:24.873490 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:24.873594 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.873600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.874449 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.874454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.875038 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:24.875047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.875562 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.875565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:24.875656 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.876211 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:24.876223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:24.878132 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":86,"threads_started":1}
W20260928 23:55:24.878392 30025 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:24.872570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:43389 (local address 127.20.88.62:42667)
0928 23:55:24.872754 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:24.872759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:24.872903 (+ 144us) server_negotiation.cc:407] Connection header received
0928 23:55:24.873077 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:24.873080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:24.873139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:24.873233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:24.873723 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.874296 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.875164 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:24.875401 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:24.875544 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:24.878272 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:43389: BlockingRecv error: recv got EOF from 127.20.88.3:43389 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:25.477928 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:25.472224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:35125)
0928 23:55:25.472608 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:25.472623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:25.472723 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:25.473009 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:25.473011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:25.473198 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:25.473338 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.473344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.474121 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.474124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.474723 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.474733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.475191 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.475193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.475276 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.475780 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:25.475791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:25.477801 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
W20260928 23:55:25.478183 30029 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:25.472314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:35125 (local address 127.20.88.62:42667)
0928 23:55:25.472649 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:25.472652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:25.472676 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:55:25.472819 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:25.472821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:25.472865 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:25.472944 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:25.473450 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.474009 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.474854 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.475086 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.475168 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.477994 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:35125: BlockingRecv error: recv got EOF from 127.20.88.1:35125 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":146,"threads_started":1}
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:25.679343 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:25.673257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:56781)
0928 23:55:25.673684 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:25.673697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:25.673845 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:25.674190 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:25.674196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:25.674418 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:25.674518 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.674525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.675479 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.675482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.676046 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.676054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.676555 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.676557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.676629 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.677162 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:25.677174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:25.679220 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":125,"threads_started":1}
W20260928 23:55:25.679621 30029 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:25.673504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:56781 (local address 127.20.88.62:42667)
0928 23:55:25.673688 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:25.673693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:25.673781 (+ 88us) server_negotiation.cc:407] Connection header received
0928 23:55:25.673948 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:25.673951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:25.674031 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:25.674125 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:25.674653 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.675376 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.676200 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.676451 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.676534 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.679498 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:56781: BlockingRecv error: recv got EOF from 127.20.88.2:56781 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:25 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:25.885126 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:25.879372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:55621)
0928 23:55:25.879701 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:25.879719 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:25.879851 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:25.880160 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:25.880163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:25.880447 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:25.880574 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.880581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.881381 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.881384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.881948 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:25.881956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.882415 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.882418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:25.882482 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.883070 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:25.883091 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:25.885009 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":137,"threads_started":1}
W20260928 23:55:25.885344 30029 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:25.879474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:55621 (local address 127.20.88.62:42667)
0928 23:55:25.879628 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:25.879633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:25.879839 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:55:25.879960 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:25.879965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:25.880032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:25.880099 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:25.880703 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.881247 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.882051 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:25.882314 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:25.882394 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:25.885156 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:55621: BlockingRecv error: recv got EOF from 127.20.88.3:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:26.484900 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:26.478980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:41395)
0928 23:55:26.479292 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:26.479307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:26.479402 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:26.479669 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:26.479671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:26.479852 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:26.479951 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.479956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.480929 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.480934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.481526 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.481536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.482023 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.482027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.482117 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.482705 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:26.482718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:26.484673 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
W20260928 23:55:26.484962 30033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:26.479148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:41395 (local address 127.20.88.62:42667)
0928 23:55:26.479429 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:26.479432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:26.479447 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:26.479494 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:26.479496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:26.479540 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:26.479633 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:26.480076 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.480797 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.481659 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.481900 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.482016 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.484829 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:41395: BlockingRecv error: recv got EOF from 127.20.88.1:41395 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:26.686674 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:26.680339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:35481)
0928 23:55:26.680659 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:26.680677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:26.680834 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:26.681122 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:26.681124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:26.681387 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:26.681479 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.681485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.682256 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.682262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.682824 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.682832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.683293 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.683296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.683378 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.683961 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:26.683973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:26.686552 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
W20260928 23:55:26.686813 30033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:26.680414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:35481 (local address 127.20.88.62:42667)
0928 23:55:26.680652 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:26.680658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:26.680780 (+ 122us) server_negotiation.cc:407] Connection header received
0928 23:55:26.680948 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:26.680951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:26.681006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:26.681102 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:26.681608 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.682144 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.682948 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.683178 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.683287 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.686689 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:35481: BlockingRecv error: recv got EOF from 127.20.88.2:35481 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:26 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:55:26.892597 30033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:26.886416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:56701 (local address 127.20.88.62:42667)
0928 23:55:26.886572 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:26.886576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:26.886664 (+ 88us) server_negotiation.cc:407] Connection header received
0928 23:55:26.886790 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:26.886792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:26.886839 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:26.886920 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:26.887435 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.888004 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.888919 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.889140 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.889223 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.892476 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:56701: BlockingRecv error: recv got EOF from 127.20.88.3:56701 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:55:26.892619 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:26.886182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:56701)
0928 23:55:26.886595 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:26.886609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:26.886707 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:26.886967 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:26.886970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:26.887159 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:26.887299 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.887305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.888124 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.888126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.888808 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:26.888818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:26.889297 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:26.889300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:26.889364 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:26.890024 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:26.890042 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:26.892439 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":114,"threads_started":1}
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:27.492460 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:27.485955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:58359)
0928 23:55:27.486330 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:27.486346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:27.486444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:27.486806 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:27.486809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:27.487050 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:27.487176 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.487184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.488329 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.488333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.488928 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.488937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.489473 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.489477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.489552 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.490093 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:27.490105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:27.492212 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
W20260928 23:55:27.492633 30037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:27.486103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:58359 (local address 127.20.88.62:42667)
0928 23:55:27.486414 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:27.486419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:27.486440 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:55:27.486529 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:27.486532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:27.486590 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:27.486707 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:27.487312 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.488166 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.489071 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.489337 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.489433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.492459 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:58359: BlockingRecv error: recv got EOF from 127.20.88.1:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:27.694057 30038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:27.687715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:51393)
0928 23:55:27.688040 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:27.688054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:27.688160 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:27.688454 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:27.688456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:27.688645 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:27.688744 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.688750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.689601 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.689604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.690207 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.690217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.690930 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.690933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.690999 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.691715 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:27.691731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:27.693933 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
W20260928 23:55:27.694202 30037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:27.687897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:51393 (local address 127.20.88.62:42667)
0928 23:55:27.688047 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:27.688051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:27.688129 (+ 78us) server_negotiation.cc:407] Connection header received
0928 23:55:27.688278 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:27.688281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:27.688330 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:27.688413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:27.688897 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.689466 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.690348 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.690589 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.690698 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.694084 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:51393: BlockingRecv error: recv got EOF from 127.20.88.2:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:27 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:27.899930 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:27.893572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:56557)
0928 23:55:27.893846 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:27.893860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:27.893957 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:27.894248 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:27.894251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:27.894429 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:27.894526 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.894532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.895370 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.895374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.896035 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:27.896045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.896631 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.896634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:27.896731 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.897339 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:27.897351 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:27.899794 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20260928 23:55:27.900107 30037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:27.893728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:56557 (local address 127.20.88.62:42667)
0928 23:55:27.893926 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:27.893929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:27.893945 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:27.894052 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:27.894054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:27.894102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:27.894175 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:27.894647 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.895194 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.896218 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:27.896505 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:27.896595 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:27.899944 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:56557: BlockingRecv error: recv got EOF from 127.20.88.3:56557 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:28.499259 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:28.493422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:35575)
0928 23:55:28.493710 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:28.493724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:28.493816 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:28.494073 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:28.494075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:28.494262 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:28.494394 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.494400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.495158 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.495160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.495730 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.495737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.496280 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.496283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.496381 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.496976 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:28.496989 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:28.499128 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20260928 23:55:28.499588 30041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:28.493488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:35575 (local address 127.20.88.62:42667)
0928 23:55:28.493759 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:28.493780 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:28.493794 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:55:28.493908 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:28.493910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:28.493956 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:28.494036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:28.494502 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.495059 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.495877 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.496128 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.496215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.499437 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:35575: BlockingRecv error: recv got EOF from 127.20.88.1:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:28.700986 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:28.695031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:38291)
0928 23:55:28.695355 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:28.695369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:28.695470 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:28.695781 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:28.695784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:28.695990 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:28.696091 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.696097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.696959 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.696962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.697532 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.697540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.698125 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.698129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.698263 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.698917 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:28.698929 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:28.700864 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":143,"threads_started":1}
W20260928 23:55:28.701216 30041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:28.695173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38291 (local address 127.20.88.62:42667)
0928 23:55:28.695336 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:28.695340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:28.695444 (+ 104us) server_negotiation.cc:407] Connection header received
0928 23:55:28.695556 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:28.695558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:28.695603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:28.695688 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:28.696211 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.696798 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.697677 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.697985 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.698071 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.701053 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:38291: BlockingRecv error: recv got EOF from 127.20.88.2:38291 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:28 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:28.918867 30043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:28.900866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:43091)
0928 23:55:28.901138 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:28.901151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:28.901290 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:28.901541 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:28.901543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:28.901722 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:28.901817 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.901822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.902717 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.902721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.903291 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:28.903300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.903747 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.903750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:28.903813 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.916410 (+ 12597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:28.916427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:28.918740 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
W20260928 23:55:28.919071 30041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:28.901025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:43091 (local address 127.20.88.62:42667)
0928 23:55:28.901321 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:28.901324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:28.901340 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:28.901383 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:28.901385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:28.901431 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:28.901503 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:28.901996 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.902597 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.903412 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:28.903640 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:28.903730 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:28.918920 (+ 15190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:43091: BlockingRecv error: recv got EOF from 127.20.88.3:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":210}
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:29.506366 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:29.500134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:60069)
0928 23:55:29.500521 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:29.500535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:29.500658 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:29.500921 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:29.500924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:29.501111 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:29.501224 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.501230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.502112 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.502115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.502698 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.502707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.503228 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.503230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.503325 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.503931 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:29.503944 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:29.506200 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":191,"threads_started":1}
W20260928 23:55:29.506610 30044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:29.500252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:60069 (local address 127.20.88.62:42667)
0928 23:55:29.500511 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:29.500515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:29.500595 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:55:29.500728 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:29.500731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:29.500773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:29.500855 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:29.501393 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.501959 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.502844 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.503097 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.503186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.506442 (+ 3256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:60069: BlockingRecv error: recv got EOF from 127.20.88.1:60069 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:29.708384 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:29.701971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:36323)
0928 23:55:29.702348 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:29.702362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:29.702470 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:29.702735 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:29.702737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:29.702923 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:29.703017 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.703022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.703791 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.703794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.704508 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.704519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.705058 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.705065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.705157 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.705897 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:29.705913 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:29.708190 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
W20260928 23:55:29.708525 30044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:29.702209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:36323 (local address 127.20.88.62:42667)
0928 23:55:29.702347 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:29.702351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:29.702417 (+ 66us) server_negotiation.cc:407] Connection header received
0928 23:55:29.702569 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:29.702572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:29.702618 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:29.702691 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:29.703127 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.703678 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.704662 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.704909 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.705006 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.708401 (+ 3395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:36323: BlockingRecv error: recv got EOF from 127.20.88.2:36323 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:29 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:29.925693 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:29.919818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:52675)
0928 23:55:29.920125 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:29.920148 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:29.920304 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:29.920564 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:29.920566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:29.920751 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:29.920853 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.920858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.921704 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.921708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.922300 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:29.922309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.922765 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.922768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:29.922838 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.923454 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:29.923466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:29.925518 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
W20260928 23:55:29.925794 30044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:29.919920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:52675 (local address 127.20.88.62:42667)
0928 23:55:29.920099 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:29.920102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:29.920277 (+ 175us) server_negotiation.cc:407] Connection header received
0928 23:55:29.920385 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:29.920387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:29.920440 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:29.920519 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:29.920993 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.921568 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.922423 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:29.922649 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:29.922733 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:29.925660 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:52675: BlockingRecv error: recv got EOF from 127.20.88.3:52675 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:30.513381 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:30.507306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:39303)
0928 23:55:30.507656 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:30.507673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:30.507826 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:30.508124 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:30.508127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:30.508432 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:30.508571 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.508578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.509382 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.509387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.509975 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.509983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.510474 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.510477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.510561 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.511157 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:30.511173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:30.513202 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
W20260928 23:55:30.513535 30049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:30.507513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:39303 (local address 127.20.88.62:42667)
0928 23:55:30.507871 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:30.507875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:30.507903 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:55:30.507942 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:30.507944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:30.507987 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:30.508093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:30.508704 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.509261 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.510109 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.510354 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.510470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.513379 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:39303: BlockingRecv error: recv got EOF from 127.20.88.1:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":68,"threads_started":1}
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:30.715701 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:30.709400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:52719)
0928 23:55:30.709752 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:30.709769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:30.709897 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:30.710209 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:30.710212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:30.710409 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:30.710507 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.710512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.711335 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.711338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.711947 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.711957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.712554 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.712558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.712630 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.713267 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:30.713282 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:30.715579 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
W20260928 23:55:30.715862 30049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:30.709469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:52719 (local address 127.20.88.62:42667)
0928 23:55:30.709644 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:30.709649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:30.709860 (+ 211us) server_negotiation.cc:407] Connection header received
0928 23:55:30.709985 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:30.709987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:30.710034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:30.710106 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:30.710637 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.711188 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.712093 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.712400 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.712508 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.715739 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:52719: BlockingRecv error: recv got EOF from 127.20.88.2:52719 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:30 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.3@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:30.932353 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:30.926632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.3:44257)
0928 23:55:30.926936 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:30.926958 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:30.927058 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:30.927315 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:30.927317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:30.927493 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:30.927591 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.927596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.928399 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.928402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.928999 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:30.929009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.929459 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.929462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:30.929526 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.930128 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:30.930140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:30.932090 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":139,"threads_started":1}
W20260928 23:55:30.932366 30049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:30.926716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44257 (local address 127.20.88.62:42667)
0928 23:55:30.926858 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:30.926862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:30.927027 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:55:30.927142 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:30.927144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:30.927194 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:30.927281 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:30.927691 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.928254 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.929114 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:30.929352 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:30.929438 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:30.932256 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.3:44257: BlockingRecv error: recv got EOF from 127.20.88.3:44257 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 23:55:31 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:31 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.1@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:31.520769 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:31.514623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.1:57285)
0928 23:55:31.514994 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:31.515008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:31.515192 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:31.515607 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:31.515611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:31.515836 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:31.515994 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:31.516004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.516898 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.516901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:31.517498 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:31.517507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.518060 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.518063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:31.518134 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:31.518679 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:31.518691 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:31.520638 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":108,"threads_started":1}
W20260928 23:55:31.521031 30053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:31.514716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57285 (local address 127.20.88.62:42667)
0928 23:55:31.515110 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:31.515114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:31.515126 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:55:31.515357 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:31.515360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:31.515425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:31.515538 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:31.516161 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.516739 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.517653 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.517906 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.517993 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:31.520877 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.1:57285: BlockingRecv error: recv got EOF from 127.20.88.1:57285 (error 108)
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":95,"threads_started":1}
Sep 28 23:55:31 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:55:31 dist-test-slave-wvpw krb5kdc[28994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.88.2@KRBTEST.COM for oryx/127.20.88.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:55:31.722784 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:31.716788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42667 (local address 127.20.88.2:58379)
0928 23:55:31.717099 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:31.717113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:31.717201 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:31.717481 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:31.717483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:31.717668 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:31.717789 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:31.717795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.718553 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.718556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:31.719140 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:31.719148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.719625 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.719628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:31.719694 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:31.720379 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:31.720391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:31.722604 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.88.62:42667: Server oryx/127.20.88.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
W20260928 23:55:31.722877 30053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:31.716990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:58379 (local address 127.20.88.62:42667)
0928 23:55:31.717135 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:31.717142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:31.717162 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:55:31.717308 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:31.717311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:31.717358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:31.717438 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:31.717902 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.718439 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.719279 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:31.719506 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:31.719599 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:31.722740 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.88.2:58379: BlockingRecv error: recv got EOF from 127.20.88.2:58379 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:55:31.793757 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29577
I20260928 23:55:31.800132 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29729
I20260928 23:55:31.805882 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29861
I20260928 23:55:31.811486 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 29507
2026-09-28T23:55:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18105 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T23:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:31Z Disabled control of system clock
I20260928 23:55:31.847043 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:39445
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:55:31.965727 30058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:31.966045 30058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:31.966081 30058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:31.970351 30058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:31.970436 30058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:31.970453 30058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:31.970469 30058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:31.970485 30058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:31.975682 30058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:39445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:39445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:31.976971 30058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:31.978368 30058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:31.987911 30066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:31.987876 30063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:31.987929 30058 server_base.cc:1034] running on GCE node
W20260928 23:55:31.987929 30064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:31.988867 30058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:31.990163 30058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:31.991420 30058 hybrid_clock.cc:648] HybridClock initialized: now 1790639731991384 us; error 55 us; skew 500 ppm
I20260928 23:55:31.993712 30058 webserver.cc:460] Webserver started at http://127.20.88.62:40963/ using document root <none> and password file <none>
I20260928 23:55:31.994375 30058 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:31.994478 30058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:31.994714 30058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:31.997100 30058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "e5aba3b2f2384a8285b8b7c0bba5520c"
format_stamp: "Formatted at 2026-09-28 23:55:31 on dist-test-slave-wvpw"
server_key: "70b8691f32b508a99d0ca38f1558d8e9"
server_key_iv: "d9e779653c703fe62c25d97de4b913f1"
server_key_version: "encryptionkey@0"
I20260928 23:55:31.997749 30058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "e5aba3b2f2384a8285b8b7c0bba5520c"
format_stamp: "Formatted at 2026-09-28 23:55:31 on dist-test-slave-wvpw"
server_key: "70b8691f32b508a99d0ca38f1558d8e9"
server_key_iv: "d9e779653c703fe62c25d97de4b913f1"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.002017 30058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:32.005090 30072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.006651 30058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:32.006835 30058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "e5aba3b2f2384a8285b8b7c0bba5520c"
format_stamp: "Formatted at 2026-09-28 23:55:31 on dist-test-slave-wvpw"
server_key: "70b8691f32b508a99d0ca38f1558d8e9"
server_key_iv: "d9e779653c703fe62c25d97de4b913f1"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.006999 30058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:32.024454 30058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:32.025244 30058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:32.025456 30058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:32.033028 30058 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:39445
I20260928 23:55:32.033057 30124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:39445 every 8 connection(s)
I20260928 23:55:32.034245 30058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:32.037887 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:32.043741 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30058
I20260928 23:55:32.043859 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:32.044104 30125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: Bootstrap starting.
I20260928 23:55:32.044165 20832 external_mini_cluster.cc:1428] Setting key 5a924335189f2283b72689a53f72f2c3
I20260928 23:55:32.046875 30125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:32.047724 30125 log.cc:824] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:32.050312 30125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: No bootstrap required, opened a new log
I20260928 23:55:32.054776 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.045726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.20.88.62:39445)
0928 23:55:32.046427 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.046443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.046487 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:55:32.047264 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.047286 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.047752 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.048117 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.048794 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.049836 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.051133 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.051646 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.051778 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.052792 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:32.052797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:32.052814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:32.053244 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:32.053305 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:32.053327 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:32.053412 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:32.053589 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 23:55:32.053941 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":154,"threads_started":1}
I20260928 23:55:32.054874 30125 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } }
I20260928 23:55:32.055238 30125 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:32.055322 30125 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5aba3b2f2384a8285b8b7c0bba5520c, State: Initialized, Role: FOLLOWER
I20260928 23:55:32.055804 30125 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } }
I20260928 23:55:32.055963 30125 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:32.056053 30125 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:32.056172 30125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:32.057267 30125 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } }
I20260928 23:55:32.057657 30125 leader_election.cc:302] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5aba3b2f2384a8285b8b7c0bba5520c; no voters:
I20260928 23:55:32.058064 30125 leader_election.cc:288] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:32.058250 30130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:32.058591 30130 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [term 1 LEADER]: Becoming Leader. State: Replica: e5aba3b2f2384a8285b8b7c0bba5520c, State: Running, Role: LEADER
I20260928 23:55:32.058982 30130 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } }
I20260928 23:55:32.059437 30125 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:32.060694 30131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } } }
I20260928 23:55:32.060825 30131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:32.060952 30132 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5aba3b2f2384a8285b8b7c0bba5520c. Latest consensus state: current_term: 1 leader_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5aba3b2f2384a8285b8b7c0bba5520c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39445 } } }
I20260928 23:55:32.061117 30132 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c [sys.catalog]: This master's current role is: LEADER
W20260928 23:55:32.064963 30145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:55:32.065070 30145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:55:32.065172 30140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:32.066094 30140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:32.072904 30140 catalog_manager.cc:1347] Generated new cluster ID: ae6eae0acba74556bd3cdda417850877
I20260928 23:55:32.073004 30140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:32.084578 30140 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:32.085995 30140 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:32.098061 30140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5aba3b2f2384a8285b8b7c0bba5520c: Generated new TSK 0
I20260928 23:55:32.099053 30140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:55:32.110253 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--builtin_ntp_servers=127.20.88.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:55:32.234174 30149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:32.234576 30149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:32.234654 30149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:32.239444 30149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:32.239567 30149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:32.239674 30149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:32.245203 30149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:32.246436 30149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:32.247893 30149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:32.258420 30154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.258462 30157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.258421 30155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.258750 30149 server_base.cc:1034] running on GCE node
I20260928 23:55:32.259320 30149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:32.260041 30149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:32.261215 30149 hybrid_clock.cc:648] HybridClock initialized: now 1790639732261191 us; error 36 us; skew 500 ppm
I20260928 23:55:32.263324 30149 webserver.cc:460] Webserver started at http://127.20.88.1:43597/ using document root <none> and password file <none>
I20260928 23:55:32.263958 30149 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:32.264065 30149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:32.264418 30149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:32.266625 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "8b82f016a8ad40c99c5f18ac673c3423"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "ff1abdee816f3339d41ef4f76a1d0bf0"
server_key_iv: "4870009070e6fd991ea6a5352b254340"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.267266 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "8b82f016a8ad40c99c5f18ac673c3423"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "ff1abdee816f3339d41ef4f76a1d0bf0"
server_key_iv: "4870009070e6fd991ea6a5352b254340"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.271390 30149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:32.274196 30163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.275549 30149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:55:32.275701 30149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "8b82f016a8ad40c99c5f18ac673c3423"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "ff1abdee816f3339d41ef4f76a1d0bf0"
server_key_iv: "4870009070e6fd991ea6a5352b254340"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.275837 30149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:32.299962 30149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:32.300820 30149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:32.301038 30149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:32.301672 30149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:32.302682 30149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:32.302754 30149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.302824 30149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:32.302867 30149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.312896 30149 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:43033
I20260928 23:55:32.312920 30276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:43033 every 8 connection(s)
I20260928 23:55:32.313979 30149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:32.317977 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30149
I20260928 23:55:32.318104 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:32.318511 20832 external_mini_cluster.cc:1428] Setting key d53097c4ab451913fe34dedd403721da
I20260928 23:55:32.320966 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--builtin_ntp_servers=127.20.88.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:55:32.325403 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.315964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:45675 (local address 127.20.88.62:39445)
0928 23:55:32.316139 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.316145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.317358 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 23:55:32.319247 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.319251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.319325 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.319416 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.320815 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.321526 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.323081 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.323432 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.323554 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.324840 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:32.324844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:32.324847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:32.324863 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:32.324893 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:32.324902 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:32.324990 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:32.325174 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:55:32.325246 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:55:32.326386 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.316494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:39445 (local address 127.20.88.1:45675)
0928 23:55:32.317193 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:32.317239 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:32.318986 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:32.319609 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:32.319617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:32.319714 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:32.320619 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.320641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.321670 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.321676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.322925 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.322936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.323612 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.323615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.323787 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.324527 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:32.324529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:32.324576 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:32.324653 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:32.324992 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:32.325004 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:32.325013 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:55:32.325156 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 23:55:32.325507 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260928 23:55:32.328044 30277 heartbeater.cc:347] Connected to a master server at 127.20.88.62:39445
I20260928 23:55:32.328397 30277 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:32.329036 30277 heartbeater.cc:511] Master 127.20.88.62:39445 requested a full tablet report, sending...
I20260928 23:55:32.331079 30089 ts_manager.cc:194] Registered new tserver with Master: 8b82f016a8ad40c99c5f18ac673c3423 (127.20.88.1:43033)
I20260928 23:55:32.332870 30089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.1:45675
W20260928 23:55:32.439710 30280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:32.439983 30280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:32.440021 30280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:32.444329 30280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:32.444392 30280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:32.444446 30280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:32.450488 30280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:32.451601 30280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:32.453011 30280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:32.463589 30286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.463589 30285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.463572 30288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.463986 30280 server_base.cc:1034] running on GCE node
I20260928 23:55:32.464483 30280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:32.465178 30280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:32.466377 30280 hybrid_clock.cc:648] HybridClock initialized: now 1790639732466353 us; error 38 us; skew 500 ppm
I20260928 23:55:32.468539 30280 webserver.cc:460] Webserver started at http://127.20.88.2:33631/ using document root <none> and password file <none>
I20260928 23:55:32.469172 30280 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:32.469282 30280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:32.469524 30280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:32.471650 30280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "3c8d03289ef349d98c2a9b9816abbbe7"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "56b4095f4622ef6ce59e24136959f75e"
server_key_iv: "4bc05188b275e9302b3cc2911e5af8e8"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.472287 30280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "3c8d03289ef349d98c2a9b9816abbbe7"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "56b4095f4622ef6ce59e24136959f75e"
server_key_iv: "4bc05188b275e9302b3cc2911e5af8e8"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.476049 30280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:55:32.478801 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.480337 30280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:32.480451 30280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "3c8d03289ef349d98c2a9b9816abbbe7"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "56b4095f4622ef6ce59e24136959f75e"
server_key_iv: "4bc05188b275e9302b3cc2911e5af8e8"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.480575 30280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:32.492223 30280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:32.493119 30280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:32.493294 30280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:32.493862 30280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:32.494843 30280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:32.494892 30280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.494935 30280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:32.494951 30280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.504593 30280 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:36327
I20260928 23:55:32.504627 30407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:36327 every 8 connection(s)
I20260928 23:55:32.505617 30280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:32.508093 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30280
I20260928 23:55:32.508211 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:32.508612 20832 external_mini_cluster.cc:1428] Setting key 7c9e23756c08c546cfb40e394373dd74
I20260928 23:55:32.511237 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--builtin_ntp_servers=127.20.88.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:55:32.516820 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.507659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50065 (local address 127.20.88.62:39445)
0928 23:55:32.507809 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.507813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.509378 (+ 1565us) server_negotiation.cc:407] Connection header received
0928 23:55:32.511397 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.511400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.511565 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.511686 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.512829 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.513410 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.514638 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.514871 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.514990 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.516206 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:32.516210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:32.516212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:32.516282 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:32.516304 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:32.516314 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:32.516398 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:32.516573 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 23:55:32.516646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:55:32.517804 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.508198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:39445 (local address 127.20.88.2:50065)
0928 23:55:32.509222 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:32.509282 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:32.511122 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:32.511934 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:32.511942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:32.512043 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:32.512664 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.512679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.513557 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.513566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.514500 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.514512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.515041 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.515045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.515226 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.515910 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:32.515912 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:32.515957 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:32.516033 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:32.516412 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:32.516424 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:32.516435 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 23:55:32.516573 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 23:55:32.516941 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":172,"threads_started":1}
I20260928 23:55:32.519255 30408 heartbeater.cc:347] Connected to a master server at 127.20.88.62:39445
I20260928 23:55:32.519523 30408 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:32.520117 30408 heartbeater.cc:511] Master 127.20.88.62:39445 requested a full tablet report, sending...
I20260928 23:55:32.521375 30089 ts_manager.cc:194] Registered new tserver with Master: 3c8d03289ef349d98c2a9b9816abbbe7 (127.20.88.2:36327)
I20260928 23:55:32.522298 30089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.2:50065
W20260928 23:55:32.629215 30411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:32.629519 30411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:32.629558 30411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:32.633731 30411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:32.633795 30411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:32.633850 30411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:32.638916 30411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:39445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:32.639992 30411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:32.641516 30411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:32.651579 30419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.651564 30411 server_base.cc:1034] running on GCE node
W20260928 23:55:32.651564 30416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.651604 30417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.652431 30411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:32.653090 30411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:32.654335 30411 hybrid_clock.cc:648] HybridClock initialized: now 1790639732654317 us; error 33 us; skew 500 ppm
I20260928 23:55:32.656461 30411 webserver.cc:460] Webserver started at http://127.20.88.3:46495/ using document root <none> and password file <none>
I20260928 23:55:32.657095 30411 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:32.657195 30411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:32.657449 30411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:32.659653 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "328541f60db148bba8741ddf219eb63b"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "897142fe305157cf90b89ad2eb09573f"
server_key_iv: "db092eb1d9398a7c25caf3efc86abce3"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.660315 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "328541f60db148bba8741ddf219eb63b"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "897142fe305157cf90b89ad2eb09573f"
server_key_iv: "db092eb1d9398a7c25caf3efc86abce3"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.664006 30411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:55:32.666778 30425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.668013 30411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:55:32.668152 30411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "328541f60db148bba8741ddf219eb63b"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "897142fe305157cf90b89ad2eb09573f"
server_key_iv: "db092eb1d9398a7c25caf3efc86abce3"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.668327 30411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:32.691078 30411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:32.691856 30411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:32.692078 30411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:32.692755 30411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:32.693751 30411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:32.693825 30411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.693899 30411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:32.693943 30411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.702741 30411 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:41537
I20260928 23:55:32.702823 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:41537 every 8 connection(s)
I20260928 23:55:32.703775 30411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:32.708433 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30411
I20260928 23:55:32.708560 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:32.708914 20832 external_mini_cluster.cc:1428] Setting key a35b68d41a7b7de5ba92b0f8c1237d15
I20260928 23:55:32.714490 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.705912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:33541 (local address 127.20.88.62:39445)
0928 23:55:32.706053 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.706057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.707205 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 23:55:32.708766 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.708770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.708837 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.708942 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.710180 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.710948 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.712396 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.712641 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.712755 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.713936 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:32.713940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:32.713942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:32.713959 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:32.713985 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:32.713992 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:32.714073 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:32.714264 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 23:55:32.714334 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:55:32.715497 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.706364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:39445 (local address 127.20.88.3:33541)
0928 23:55:32.707047 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:32.707089 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:32.708486 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:32.709155 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:32.709162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:32.709255 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:32.709971 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.709988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.711089 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.711093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.712221 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:32.712276 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:32.712811 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:32.712816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:32.713008 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:32.713615 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:32.713617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:32.713662 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:32.713737 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:32.714094 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:32.714109 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:32.714115 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:55:32.714263 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 23:55:32.714593 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260928 23:55:32.716787 30539 heartbeater.cc:347] Connected to a master server at 127.20.88.62:39445
I20260928 23:55:32.717063 30539 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:32.717643 30539 heartbeater.cc:511] Master 127.20.88.62:39445 requested a full tablet report, sending...
I20260928 23:55:32.718798 30089 ts_manager.cc:194] Registered new tserver with Master: 328541f60db148bba8741ddf219eb63b (127.20.88.3:41537)
I20260928 23:55:32.719380 30089 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.3:33541
I20260928 23:55:32.723066 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:55:32.726425 30128 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:32.725322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.20.88.62:39445)
0928 23:55:32.725514 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.725519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.725701 (+ 182us) server_negotiation.cc:407] Connection header received
0928 23:55:32.725881 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.725884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.725940 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.726028 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.726307 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59210: BlockingRecv error: recv got EOF from 127.0.0.1:59210 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:55:32.726656 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30149
I20260928 23:55:32.732224 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30280
I20260928 23:55:32.737884 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30411
I20260928 23:55:32.743531 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30058
2026-09-28T23:55:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (928 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T23:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:32Z Disabled control of system clock
I20260928 23:55:32.774813 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:38135
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:55:32.889178 30550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:32.889494 30550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:32.889534 30550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:32.893743 30550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:32.893817 30550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:32.893833 30550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:32.893851 30550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:32.893867 30550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:32.898825 30550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:38135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:32.899964 30550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:32.901448 30550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:32.910923 30558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.911129 30550 server_base.cc:1034] running on GCE node
W20260928 23:55:32.910924 30556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:32.911034 30555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:32.911933 30550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:32.913236 30550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:32.914438 30550 hybrid_clock.cc:648] HybridClock initialized: now 1790639732914397 us; error 59 us; skew 500 ppm
I20260928 23:55:32.916553 30550 webserver.cc:460] Webserver started at http://127.20.88.62:40817/ using document root <none> and password file <none>
I20260928 23:55:32.917171 30550 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:32.917240 30550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:32.917418 30550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:32.919631 30550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "7ef838ea4bd2471a92f0948de9cc5c71"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "e51fb161629d01d68cee7a0d1624afb7"
server_key_iv: "c4dfc76488cf6c70145abfa22c0b16e5"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.920472 30550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "7ef838ea4bd2471a92f0948de9cc5c71"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "e51fb161629d01d68cee7a0d1624afb7"
server_key_iv: "c4dfc76488cf6c70145abfa22c0b16e5"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.924731 30550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:32.927537 30564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:32.929193 30550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:32.929328 30550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "7ef838ea4bd2471a92f0948de9cc5c71"
format_stamp: "Formatted at 2026-09-28 23:55:32 on dist-test-slave-wvpw"
server_key: "e51fb161629d01d68cee7a0d1624afb7"
server_key_iv: "c4dfc76488cf6c70145abfa22c0b16e5"
server_key_version: "encryptionkey@0"
I20260928 23:55:32.929448 30550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:32.934878 30550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:32.935580 30550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:32.935743 30550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:32.943607 30550 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:38135
I20260928 23:55:32.943634 30616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:38135 every 8 connection(s)
I20260928 23:55:32.944734 30550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:32.948437 30617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:32.951741 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30550
I20260928 23:55:32.951928 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:32.952215 20832 external_mini_cluster.cc:1428] Setting key cf359b4b48b72bfca6c450273c0e859d
I20260928 23:55:32.955336 30617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71: Bootstrap starting.
I20260928 23:55:32.958122 30617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:32.958416 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:32.953815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.20.88.62:38135)
0928 23:55:32.954492 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:32.954509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:32.954547 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:55:32.955213 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:32.955231 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:32.955661 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:32.955992 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:32.956975 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:32.956981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:32.956994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:32.957006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:32.957068 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:32.957085 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:32.957211 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:32.957310 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:55:32.957607 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260928 23:55:32.959100 30617 log.cc:824] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:32.961915 30617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71: No bootstrap required, opened a new log
I20260928 23:55:32.965914 30617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } }
I20260928 23:55:32.966187 30617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:32.966238 30617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef838ea4bd2471a92f0948de9cc5c71, State: Initialized, Role: FOLLOWER
I20260928 23:55:32.966691 30617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } }
I20260928 23:55:32.966823 30617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:32.966867 30617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:32.966949 30617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:32.967962 30617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } }
I20260928 23:55:32.968364 30617 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef838ea4bd2471a92f0948de9cc5c71; no voters:
I20260928 23:55:32.968677 30617 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:32.968933 30622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:32.969216 30622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [term 1 LEADER]: Becoming Leader. State: Replica: 7ef838ea4bd2471a92f0948de9cc5c71, State: Running, Role: LEADER
I20260928 23:55:32.969559 30622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } }
I20260928 23:55:32.970084 30617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:32.971549 30623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } } }
I20260928 23:55:32.971591 30624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef838ea4bd2471a92f0948de9cc5c71. Latest consensus state: current_term: 1 leader_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef838ea4bd2471a92f0948de9cc5c71" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 38135 } } }
I20260928 23:55:32.971685 30623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:32.971693 30624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:32.972869 30632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:32.976030 30632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:32.982147 30632 catalog_manager.cc:1347] Generated new cluster ID: c4516e13a4074a1da65c6f67189d7de9
I20260928 23:55:32.982244 30632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:33.010417 30632 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:33.010838 30632 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:33.020112 30632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef838ea4bd2471a92f0948de9cc5c71: Generated new TSK 0
I20260928 23:55:33.021000 30632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:55:33.026171 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--builtin_ntp_servers=127.20.88.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:55:33.155799 30641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:33.156131 30641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:33.156183 30641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:33.160566 30641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:33.160650 30641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:33.160712 30641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:33.166018 30641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:33.167363 30641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:33.168962 30641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:33.180032 30646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.180099 30647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.180032 30649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:33.180223 30641 server_base.cc:1034] running on GCE node
I20260928 23:55:33.180961 30641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:33.181766 30641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:33.182914 30641 hybrid_clock.cc:648] HybridClock initialized: now 1790639733182887 us; error 42 us; skew 500 ppm
I20260928 23:55:33.184985 30641 webserver.cc:460] Webserver started at http://127.20.88.1:44217/ using document root <none> and password file <none>
I20260928 23:55:33.185618 30641 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:33.185688 30641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:33.185916 30641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:33.188486 30641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "678bca761b5245a5951cf9554629909c"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "ac0e9a6a0d31547688ab801a8d4b707d"
server_key_iv: "48ef5538d93ab3664d6afc1dd92a6c6b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.189100 30641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "678bca761b5245a5951cf9554629909c"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "ac0e9a6a0d31547688ab801a8d4b707d"
server_key_iv: "48ef5538d93ab3664d6afc1dd92a6c6b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.193578 30641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:33.196759 30655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.198137 30641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:55:33.198325 30641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "678bca761b5245a5951cf9554629909c"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "ac0e9a6a0d31547688ab801a8d4b707d"
server_key_iv: "48ef5538d93ab3664d6afc1dd92a6c6b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.198465 30641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:33.203564 30641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:33.204301 30641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:33.204489 30641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:33.205080 30641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:33.206067 30641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:33.206133 30641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.206211 30641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:33.206265 30641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.215585 30641 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:37765
I20260928 23:55:33.215610 30768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:37765 every 8 connection(s)
I20260928 23:55:33.216645 30641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:33.223467 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.218684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:58725 (local address 127.20.88.62:38135)
0928 23:55:33.218844 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:33.218848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:33.219961 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 23:55:33.221423 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:33.221427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:33.221476 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:33.221588 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:33.222929 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:33.222933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:33.222936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:33.222954 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:33.222976 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:33.222984 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:33.223084 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:33.223276 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 23:55:33.223326 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:55:33.223786 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30641
I20260928 23:55:33.223905 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:33.224195 20832 external_mini_cluster.cc:1428] Setting key 8624b040271b7e5ca281aa30a7615a57
I20260928 23:55:33.224678 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.219094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:38135 (local address 127.20.88.1:58725)
0928 23:55:33.219782 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:33.219824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:33.221184 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:33.221778 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:33.221785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:33.221882 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:33.222634 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:33.222636 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:33.222680 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:33.222759 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:33.223128 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:33.223138 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:33.223143 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:55:33.223336 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 23:55:33.223616 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260928 23:55:33.226008 30769 heartbeater.cc:347] Connected to a master server at 127.20.88.62:38135
I20260928 23:55:33.226290 30769 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:33.226466 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--builtin_ntp_servers=127.20.88.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:55:33.226768 30769 heartbeater.cc:511] Master 127.20.88.62:38135 requested a full tablet report, sending...
I20260928 23:55:33.228740 30581 ts_manager.cc:194] Registered new tserver with Master: 678bca761b5245a5951cf9554629909c (127.20.88.1:37765)
W20260928 23:55:33.348711 30772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:33.349077 30772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:33.349123 30772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:33.353513 30772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:33.353585 30772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:33.353646 30772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:33.358724 30772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:33.359982 30772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:33.361672 30772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:33.371731 30777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.371731 30780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.371848 30778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:33.372116 30772 server_base.cc:1034] running on GCE node
I20260928 23:55:33.372635 30772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:33.373306 30772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:33.374480 30772 hybrid_clock.cc:648] HybridClock initialized: now 1790639733374445 us; error 48 us; skew 500 ppm
I20260928 23:55:33.376662 30772 webserver.cc:460] Webserver started at http://127.20.88.2:44211/ using document root <none> and password file <none>
I20260928 23:55:33.377249 30772 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:33.377318 30772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:33.377490 30772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:33.379555 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "04131711850d4df2a9e39a4cfdac693e"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "0278b6221348749e2729f117547afcb9"
server_key_iv: "860eab63ecbb32befc74c2489719b79b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.380096 30772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "04131711850d4df2a9e39a4cfdac693e"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "0278b6221348749e2729f117547afcb9"
server_key_iv: "860eab63ecbb32befc74c2489719b79b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.383999 30772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 23:55:33.386806 30786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.388226 30772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:55:33.388470 30772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "04131711850d4df2a9e39a4cfdac693e"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "0278b6221348749e2729f117547afcb9"
server_key_iv: "860eab63ecbb32befc74c2489719b79b"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.388602 30772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:33.393330 30772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:33.393998 30772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:33.394197 30772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:33.394778 30772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:33.395735 30772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:33.395782 30772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.395846 30772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:33.395880 30772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.405200 30772 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:41525
I20260928 23:55:33.405216 30899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:41525 every 8 connection(s)
I20260928 23:55:33.406201 30772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:33.412881 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.408146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50195 (local address 127.20.88.62:38135)
0928 23:55:33.408359 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:33.408363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:33.409491 (+ 1128us) server_negotiation.cc:407] Connection header received
0928 23:55:33.410971 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:33.410974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:33.411017 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:33.411099 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:33.412401 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:33.412404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:33.412406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:33.412420 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:33.412440 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:33.412448 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:33.412539 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:33.412720 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 23:55:33.412766 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:55:33.413687 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30772
I20260928 23:55:33.413825 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:33.414177 20832 external_mini_cluster.cc:1428] Setting key 28529c0839625eb40d03db3d7e50d693
I20260928 23:55:33.414331 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.408629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:38135 (local address 127.20.88.2:50195)
0928 23:55:33.409324 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:33.409370 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:33.410708 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:33.411297 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:33.411305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:33.411408 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:33.412084 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:33.412086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:33.412129 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:33.412212 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:33.412563 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:33.412572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:33.412578 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:55:33.412760 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 23:55:33.413050 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260928 23:55:33.415657 30900 heartbeater.cc:347] Connected to a master server at 127.20.88.62:38135
I20260928 23:55:33.415926 30900 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:33.416412 30900 heartbeater.cc:511] Master 127.20.88.62:38135 requested a full tablet report, sending...
I20260928 23:55:33.416744 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--builtin_ntp_servers=127.20.88.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:55:33.417526 30580 ts_manager.cc:194] Registered new tserver with Master: 04131711850d4df2a9e39a4cfdac693e (127.20.88.2:41525)
W20260928 23:55:33.538113 30903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:33.538434 30903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:33.538479 30903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:33.542682 30903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:33.542768 30903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:33.542833 30903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:33.548050 30903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:38135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:33.549289 30903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:33.550755 30903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:33.561388 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.561362 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:33.561359 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:33.561666 30903 server_base.cc:1034] running on GCE node
I20260928 23:55:33.562307 30903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:33.563014 30903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:33.564178 30903 hybrid_clock.cc:648] HybridClock initialized: now 1790639733564130 us; error 65 us; skew 500 ppm
I20260928 23:55:33.566191 30903 webserver.cc:460] Webserver started at http://127.20.88.3:35849/ using document root <none> and password file <none>
I20260928 23:55:33.566771 30903 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:33.566833 30903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:33.567008 30903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:33.569278 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "0aab765171184c0aa916650770950cff"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "67d0fb07712d8ff74463102d62b021c0"
server_key_iv: "d621f6d375bc70415527ffb56740be0e"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.569834 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "0aab765171184c0aa916650770950cff"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "67d0fb07712d8ff74463102d62b021c0"
server_key_iv: "d621f6d375bc70415527ffb56740be0e"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.574265 30903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:55:33.577227 30917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.578648 30903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:33.578822 30903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "0aab765171184c0aa916650770950cff"
format_stamp: "Formatted at 2026-09-28 23:55:33 on dist-test-slave-wvpw"
server_key: "67d0fb07712d8ff74463102d62b021c0"
server_key_iv: "d621f6d375bc70415527ffb56740be0e"
server_key_version: "encryptionkey@0"
I20260928 23:55:33.578959 30903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:33.584182 30903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:33.584944 30903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:33.585143 30903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:33.585839 30903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:33.587088 30903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:33.587174 30903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.587275 30903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:33.587322 30903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:33.596742 30903 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:41585
I20260928 23:55:33.596771 31030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:41585 every 8 connection(s)
I20260928 23:55:33.597769 30903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:33.604439 30620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.599686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:36995 (local address 127.20.88.62:38135)
0928 23:55:33.599833 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:33.599837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:33.600984 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 23:55:33.602438 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:33.602442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:33.602494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:33.602576 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:33.603890 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:33.603894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:33.603897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:55:33.603912 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:33.603940 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:55:33.603950 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:33.604057 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:33.604201 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:55:33.604288 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:55:33.604514 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 30903
I20260928 23:55:33.604715 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:33.605119 20832 external_mini_cluster.cc:1428] Setting key 4dfad12d5b07a5dd6e493a07489a0bea
I20260928 23:55:33.605600 31033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:33.600067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:38135 (local address 127.20.88.3:36995)
0928 23:55:33.600826 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:33.600871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:33.602192 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:33.602762 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:33.602769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:33.602866 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:33.603510 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:55:33.603512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:33.603556 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:55:33.603631 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:33.604034 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:33.604043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:33.604049 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:55:33.604274 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 23:55:33.604620 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":179,"threads_started":1}
I20260928 23:55:33.606874 31031 heartbeater.cc:347] Connected to a master server at 127.20.88.62:38135
I20260928 23:55:33.607127 31031 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:33.607582 31031 heartbeater.cc:511] Master 127.20.88.62:38135 requested a full tablet report, sending...
I20260928 23:55:33.608757 30581 ts_manager.cc:194] Registered new tserver with Master: 0aab765171184c0aa916650770950cff (127.20.88.3:41585)
I20260928 23:55:33.619426 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:55:33.622663 30620 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:55:33.621601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.20.88.62:38135)
0928 23:55:33.621765 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:33.621776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:33.621919 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:55:33.622085 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:33.622088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:33.622124 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:33.622250 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:33.622517 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57574: BlockingRecv error: recv got EOF from 127.0.0.1:57574 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:55:33.622923 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30641
I20260928 23:55:33.628826 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30772
I20260928 23:55:33.634136 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30903
I20260928 23:55:33.639685 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 30550
2026-09-28T23:55:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (895 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:33 dist-test-slave-wvpw krb5kdc[31041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:33 dist-test-slave-wvpw krb5kdc[31041](info): set up 2 sockets
Sep 28 23:55:33 dist-test-slave-wvpw krb5kdc[31041](info): commencing operation
krb5kdc: starting...
W20260928 23:55:34.701735 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:34 dist-test-slave-wvpw krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:55:34.848793 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41401
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:34.969285 31054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:34.969590 31054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:34.969632 31054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:34.973974 31054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:55:34.974054 31054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:34.974076 31054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:34.974092 31054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:55:34.974109 31054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:55:34.979753 31054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:41401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:34.981010 31054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:34.982082 31054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:34.991524 31062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:34.991582 31054 server_base.cc:1034] running on GCE node
W20260928 23:55:34.991534 31060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:34.991528 31059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:34.992561 31054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:34.993878 31054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:34.995079 31054 hybrid_clock.cc:648] HybridClock initialized: now 1790639734995040 us; error 53 us; skew 500 ppm
Sep 28 23:55:34 dist-test-slave-wvpw krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639734, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:34.998608 31054 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:55:34.999730 31054 webserver.cc:460] Webserver started at http://127.20.88.62:33587/ using document root <none> and password file <none>
I20260928 23:55:35.000411 31054 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:35.000485 31054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:35.000717 31054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:35.002898 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "dd2ed278d7c44e24938fc19d52b3b202"
format_stamp: "Formatted at 2026-09-28 23:55:34 on dist-test-slave-wvpw"
server_key: "46b250154e5d9c313724082ddca223d1"
server_key_iv: "aa366462e8d1c5df3a3ad054ba187d87"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.003507 31054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "dd2ed278d7c44e24938fc19d52b3b202"
format_stamp: "Formatted at 2026-09-28 23:55:34 on dist-test-slave-wvpw"
server_key: "46b250154e5d9c313724082ddca223d1"
server_key_iv: "aa366462e8d1c5df3a3ad054ba187d87"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.007776 31054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:35.010888 31069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.012454 31054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:55:35.012615 31054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "dd2ed278d7c44e24938fc19d52b3b202"
format_stamp: "Formatted at 2026-09-28 23:55:34 on dist-test-slave-wvpw"
server_key: "46b250154e5d9c313724082ddca223d1"
server_key_iv: "aa366462e8d1c5df3a3ad054ba187d87"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.012753 31054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:35.031891 31054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:35.035669 31054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:35.035915 31054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:35.043521 31054 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:41401
I20260928 23:55:35.043622 31121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:41401 every 8 connection(s)
I20260928 23:55:35.044620 31054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:55:35.047647 31122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:35.049881 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 31054
I20260928 23:55:35.049986 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:55:35.050294 20832 external_mini_cluster.cc:1428] Setting key 6c987a3f6477b61b1d0e2207f68809fb
I20260928 23:55:35.055017 31122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: Bootstrap starting.
I20260928 23:55:35.057866 31122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:35.058882 31122 log.cc:824] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:35.061311 31122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: No bootstrap required, opened a new log
I20260928 23:55:35.065095 31122 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } }
I20260928 23:55:35.065349 31122 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:35.065407 31122 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2ed278d7c44e24938fc19d52b3b202, State: Initialized, Role: FOLLOWER
I20260928 23:55:35.065900 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.051802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.20.88.62:41401)
0928 23:55:35.052295 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.052312 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.052349 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:55:35.053021 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.053040 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.053399 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.053699 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:35.054629 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.055701 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.056899 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.057389 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.057510 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.060611 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:35.060616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:35.060632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:35.060666 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:35.062479 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.062956 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.062967 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.062972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.063086 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.063369 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.063376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.063378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.063834 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:35.064207 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.064590 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.064926 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":139,"threads_started":1}
I20260928 23:55:35.065892 31122 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } }
I20260928 23:55:35.066053 31122 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:55:35.066109 31122 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:55:35.066211 31122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:35.067291 31122 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } }
I20260928 23:55:35.067667 31122 leader_election.cc:302] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2ed278d7c44e24938fc19d52b3b202; no voters:
I20260928 23:55:35.068013 31122 leader_election.cc:288] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:55:35.068132 31127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:35.068404 31127 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [term 1 LEADER]: Becoming Leader. State: Replica: dd2ed278d7c44e24938fc19d52b3b202, State: Running, Role: LEADER
I20260928 23:55:35.068753 31127 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } }
I20260928 23:55:35.069309 31122 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:55:35.070418 31128 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2ed278d7c44e24938fc19d52b3b202. Latest consensus state: current_term: 1 leader_uuid: "dd2ed278d7c44e24938fc19d52b3b202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } } }
I20260928 23:55:35.070509 31129 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2ed278d7c44e24938fc19d52b3b202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2ed278d7c44e24938fc19d52b3b202" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41401 } } }
I20260928 23:55:35.070712 31129 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:35.070746 31128 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202 [sys.catalog]: This master's current role is: LEADER
I20260928 23:55:35.071165 31136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:55:35.074555 31136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:55:35.080585 31136 catalog_manager.cc:1347] Generated new cluster ID: cef7590b30f8425c8fa55c27cb3f032e
I20260928 23:55:35.080683 31136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:55:35.102810 31136 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:55:35.103929 31136 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:55:35.109815 31136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: Generated new TSK 0
I20260928 23:55:35.110638 31136 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:55:35.148384 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:35.285162 31148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:35.285470 31148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:35.285518 31148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:35.289803 31148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:35.289891 31148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:35.289956 31148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:55:35.295712 31148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:35.296895 31148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:35.297931 31148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:35.308728 31154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.308729 31153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.308722 31156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:35.308792 31148 server_base.cc:1034] running on GCE node
I20260928 23:55:35.309443 31148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:35.310235 31148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:35.311455 31148 hybrid_clock.cc:648] HybridClock initialized: now 1790639735311434 us; error 36 us; skew 500 ppm
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:35.314699 31148 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:55:35.315713 31148 webserver.cc:460] Webserver started at http://127.20.88.1:42667/ using document root <none> and password file <none>
I20260928 23:55:35.316366 31148 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:35.316442 31148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:35.316620 31148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:35.319041 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "d6c3aec8a04045619a956bf01e677917"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "a6e8edfb0766697a365deafbbca17862"
server_key_iv: "bc1701e524ba98464699a92f66a30884"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.319608 31148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "d6c3aec8a04045619a956bf01e677917"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "a6e8edfb0766697a365deafbbca17862"
server_key_iv: "bc1701e524ba98464699a92f66a30884"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.323774 31148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:55:35.326767 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.328281 31148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:55:35.328425 31148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "d6c3aec8a04045619a956bf01e677917"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "a6e8edfb0766697a365deafbbca17862"
server_key_iv: "bc1701e524ba98464699a92f66a30884"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.328536 31148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:35.349907 31148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:35.353422 31148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:35.353600 31148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:35.354189 31148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:35.355000 31148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:35.355048 31148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.355088 31148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:35.355114 31148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.364277 31148 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35695
I20260928 23:55:35.364297 31276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35695 every 8 connection(s)
I20260928 23:55:35.365245 31148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:55:35.366590 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 31148
I20260928 23:55:35.366726 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:55:35.367019 20832 external_mini_cluster.cc:1428] Setting key 8cc2c7d12d4c43501c77c0d1968b5248
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:55:35.380540 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.368425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38405 (local address 127.20.88.62:41401)
0928 23:55:35.368565 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.368570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.368586 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:35.369879 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.369882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.369937 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.370015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:35.371576 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.372290 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.373487 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.373739 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.373854 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.376545 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:35.376548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:35.376550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:35.376577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:35.378112 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.378792 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.378796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.378798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.378880 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.379238 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.379243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.379245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.379494 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:35.379610 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.380201 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.380373 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:55:35.381448 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.367580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41401 (local address 127.20.88.1:38405)
0928 23:55:35.368061 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.368150 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.369623 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.370195 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.370203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.370605 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:35.371402 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.371426 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.372432 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.372434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.373355 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.373364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.373901 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.373905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.374091 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.374787 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:35.374803 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:35.376351 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:35.378259 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.378266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.378286 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.378686 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.378973 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.378977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.378979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.379145 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.379626 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:35.379635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:35.379916 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.380162 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.380554 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260928 23:55:35.382956 31277 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41401
I20260928 23:55:35.383258 31277 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:35.383930 31277 heartbeater.cc:511] Master 127.20.88.62:41401 requested a full tablet report, sending...
I20260928 23:55:35.385789 31082 ts_manager.cc:194] Registered new tserver with Master: d6c3aec8a04045619a956bf01e677917 (127.20.88.1:35695)
I20260928 23:55:35.386997 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:38405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:55:35.400841 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:35.524394 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:35.524740 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:35.524811 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:35.529120 31282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:35.529227 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:35.529328 31282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:55:35.535239 31282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:35.536507 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:35.537590 31282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:35.547876 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.547876 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.548015 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:35.548288 31282 server_base.cc:1034] running on GCE node
I20260928 23:55:35.548784 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:35.549562 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:35.550767 31282 hybrid_clock.cc:648] HybridClock initialized: now 1790639735550743 us; error 36 us; skew 500 ppm
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:35.553956 31282 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:55:35.554949 31282 webserver.cc:460] Webserver started at http://127.20.88.2:45473/ using document root <none> and password file <none>
I20260928 23:55:35.555552 31282 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:35.555642 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:35.555879 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:35.558121 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "0b679e7c29584663aae82273c5852232"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "2421f40490f8c87f5b0c771078a7b6ec"
server_key_iv: "f34a165ab9c42fd678a475cd20667903"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.558717 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "0b679e7c29584663aae82273c5852232"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "2421f40490f8c87f5b0c771078a7b6ec"
server_key_iv: "f34a165ab9c42fd678a475cd20667903"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.562847 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:55:35.565805 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.567373 31282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:55:35.567535 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "0b679e7c29584663aae82273c5852232"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "2421f40490f8c87f5b0c771078a7b6ec"
server_key_iv: "f34a165ab9c42fd678a475cd20667903"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.567679 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:35.580586 31282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:35.584146 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:35.584408 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:35.585044 31282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:35.585947 31282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:35.586032 31282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.586135 31282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:35.586176 31282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.595472 31282 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:36695
I20260928 23:55:35.595489 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:36695 every 8 connection(s)
I20260928 23:55:35.596585 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:55:35.598832 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 31282
I20260928 23:55:35.598948 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:55:35.599244 20832 external_mini_cluster.cc:1428] Setting key 0e0bde2ebad2e25571265d3a528d9cc6
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:55:35.613164 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.598710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:56029 (local address 127.20.88.62:41401)
0928 23:55:35.598873 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.598878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.599892 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 23:55:35.601482 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.601488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.601562 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.601654 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:35.602931 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.603682 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.605003 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.605258 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.605380 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.608401 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:35.608407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:35.608410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:35.608445 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:35.610219 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.610958 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.610961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.610963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.611053 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.611611 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.611614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.611616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.611861 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:35.611947 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.612898 (+ 951us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.613022 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:55:35.614558 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.599160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41401 (local address 127.20.88.2:56029)
0928 23:55:35.599722 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.599780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.601251 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.601847 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.601855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.602245 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:35.602770 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.602785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.603822 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.603825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.604862 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.604873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.605429 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.605434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.605623 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.606285 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:35.606303 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:35.608086 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:35.610378 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.610385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.610401 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.610764 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.611178 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.611182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.611184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.611424 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.611996 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:35.612003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:35.612575 (+ 572us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.612792 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.613248 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":181,"threads_started":1}
I20260928 23:55:35.615957 31411 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41401
I20260928 23:55:35.616344 31411 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:35.617058 31411 heartbeater.cc:511] Master 127.20.88.62:41401 requested a full tablet report, sending...
I20260928 23:55:35.618489 31082 ts_manager.cc:194] Registered new tserver with Master: 0b679e7c29584663aae82273c5852232 (127.20.88.2:36695)
I20260928 23:55:35.619131 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:56029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:55:35.633407 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:55:35.747604 31416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:55:35.747915 31416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:55:35.747996 31416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:55:35.752347 31416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:55:35.752435 31416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:55:35.752532 31416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:55:35.758234 31416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:55:35.759439 31416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:55:35.760613 31416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:55:35.771180 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.771235 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:55:35.771235 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:55:35.771377 31416 server_base.cc:1034] running on GCE node
I20260928 23:55:35.771840 31416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:55:35.772605 31416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:55:35.773803 31416 hybrid_clock.cc:648] HybridClock initialized: now 1790639735773783 us; error 39 us; skew 500 ppm
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:55:35.776691 31416 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:55:35.777685 31416 webserver.cc:460] Webserver started at http://127.20.88.3:37193/ using document root <none> and password file <none>
I20260928 23:55:35.778261 31416 fs_manager.cc:359] Metadata directory not provided
I20260928 23:55:35.778324 31416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:55:35.778496 31416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:55:35.780748 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "33052a024ba84cdfaf1049f7f99dc227"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "b80bd5d910391e8faa2db05548a36daa"
server_key_iv: "eb3596dafda3b490b5f36167ba2de808"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.781311 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "33052a024ba84cdfaf1049f7f99dc227"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "b80bd5d910391e8faa2db05548a36daa"
server_key_iv: "eb3596dafda3b490b5f36167ba2de808"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.785374 31416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:55:35.788097 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.789606 31416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:55:35.789762 31416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "33052a024ba84cdfaf1049f7f99dc227"
format_stamp: "Formatted at 2026-09-28 23:55:35 on dist-test-slave-wvpw"
server_key: "b80bd5d910391e8faa2db05548a36daa"
server_key_iv: "eb3596dafda3b490b5f36167ba2de808"
server_key_version: "encryptionkey@0"
I20260928 23:55:35.789891 31416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:55:35.807600 31416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:55:35.810577 31416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:55:35.810779 31416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:55:35.811378 31416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:55:35.812196 31416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:55:35.812269 31416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.812340 31416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:55:35.812379 31416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:55:35.821473 31416 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:46775
I20260928 23:55:35.821494 31544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:46775 every 8 connection(s)
I20260928 23:55:35.822382 31416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:55:35.830704 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 31416
I20260928 23:55:35.830873 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:55:35.831176 20832 external_mini_cluster.cc:1428] Setting key 9221fff33a1334a580079a7f62894780
Sep 28 23:55:35 dist-test-slave-wvpw krb5kdc[31041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:55:35.837350 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.824268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:49961 (local address 127.20.88.62:41401)
0928 23:55:35.824419 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.824423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.825323 (+ 900us) server_negotiation.cc:407] Connection header received
0928 23:55:35.826801 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.826805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.826867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.826968 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:35.828222 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.828787 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.829963 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.830193 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.830304 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.833515 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:35.833518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:35.833520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:35.833547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:35.834966 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.835636 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.835642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.835644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.835732 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.836092 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.836095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.836097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.836368 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:35.836472 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.837032 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.837182 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:55:35.838238 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.824702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41401 (local address 127.20.88.3:49961)
0928 23:55:35.825164 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.825214 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.826556 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.827162 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.827169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.827559 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:55:35.828062 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.828076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.828918 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.828927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.829835 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.829845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.830352 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.830357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.830562 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.831280 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:55:35.831306 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:55:35.833300 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:55:35.835180 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.835188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.835208 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.835503 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.835826 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:55:35.835830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:55:35.835832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:55:35.835992 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:55:35.836478 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:55:35.836489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:55:35.836786 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.836991 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.837356 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260928 23:55:35.839422 31545 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41401
I20260928 23:55:35.839658 31545 heartbeater.cc:464] Registering TS with master...
I20260928 23:55:35.840148 31545 heartbeater.cc:511] Master 127.20.88.62:41401 requested a full tablet report, sending...
I20260928 23:55:35.841393 31082 ts_manager.cc:194] Registered new tserver with Master: 33052a024ba84cdfaf1049f7f99dc227 (127.20.88.3:46775)
I20260928 23:55:35.841986 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:49961
I20260928 23:55:35.845824 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:55:35.855783 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.847952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.20.88.62:41401)
0928 23:55:35.848136 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.848142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.848359 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:55:35.848527 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.848530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.848581 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.848663 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:55:35.849415 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.850053 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.850929 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.851153 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.851261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.852719 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:55:35.852723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:55:35.852726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:55:35.852764 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:55:35.854045 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.854477 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.854481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.854483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.854567 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:55:35.854836 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:55:35.854841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:55:35.854843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:55:35.855083 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:55:35.855224 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.855463 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.855606 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 23:55:35.858978 31082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:55:35.862323 31082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:55:35.877835 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.872886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.20.88.3:46775)
0928 23:55:35.873211 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.873222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.873246 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:55:35.873320 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.873325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.873548 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.874002 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:35.874164 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.875096 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.876209 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.877092 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.877213 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.877363 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.877568 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.877668 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":128,"threads_started":1}
I20260928 23:55:35.877830 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.871555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:46775 (local address 127.0.0.1:33308)
0928 23:55:35.871898 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.871950 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.872083 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.873708 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.873712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.873736 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:35.873861 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.873867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.875225 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.875228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.876069 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.876079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.877215 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.877218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.877342 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.877382 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.877514 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.877668 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260928 23:55:35.877949 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.872134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.20.88.2:36695)
0928 23:55:35.872633 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.872642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.872662 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:55:35.872724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.872728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.872886 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.873046 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:35.873494 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.874448 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.876268 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.877111 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.877213 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.877369 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.877695 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.877793 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":159,"threads_started":1}
I20260928 23:55:35.878196 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.871985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36695 (local address 127.0.0.1:57612)
0928 23:55:35.872347 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.872361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.872449 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.873038 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.873045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.873076 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:35.873233 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.873240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.874573 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.874581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.876106 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.876116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.877423 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.877426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.877515 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.877537 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.877605 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.878077 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":397,"thread_start_us":127,"threads_started":1}
I20260928 23:55:35.879961 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.872493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.20.88.1:35695)
0928 23:55:35.872784 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.872799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.873801 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 23:55:35.874011 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.874016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.874211 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.874328 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:35.874816 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.875956 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.877414 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.879099 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.879200 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.879332 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.879690 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.879797 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":131,"threads_started":1}
I20260928 23:55:35.880429 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.873368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35695 (local address 127.0.0.1:33100)
0928 23:55:35.873708 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.873721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.873863 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.874508 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.874511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.874528 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:35.874694 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.874700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.876117 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.876121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.877280 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.877290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.879415 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.879417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.879520 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.879542 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.880183 (+ 641us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.880305 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":52,"thread_start_us":93,"threads_started":1}
I20260928 23:55:35.881510 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f (DEFAULT_TABLE table=test-table [id=0d257bb8a2bc4f9cbe6b41470f030d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:35.881510 31345 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f (DEFAULT_TABLE table=test-table [id=0d257bb8a2bc4f9cbe6b41470f030d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:35.882750 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf2fd80020242fe92f2c3c5078c870f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:35.882748 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf2fd80020242fe92f2c3c5078c870f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:35.886636 31211 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f (DEFAULT_TABLE table=test-table [id=0d257bb8a2bc4f9cbe6b41470f030d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:55:35.887697 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf2fd80020242fe92f2c3c5078c870f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:55:35.890678 31562 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Bootstrap starting.
I20260928 23:55:35.891610 31563 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Bootstrap starting.
I20260928 23:55:35.894635 31562 tablet_bootstrap.cc:654] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:35.894634 31563 tablet_bootstrap.cc:654] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:35.895596 31563 log.cc:824] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:35.895635 31562 log.cc:824] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:35.896903 31564 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: Bootstrap starting.
I20260928 23:55:35.897735 31562 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: No bootstrap required, opened a new log
I20260928 23:55:35.897851 31563 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: No bootstrap required, opened a new log
I20260928 23:55:35.897984 31562 ts_tablet_manager.cc:1397] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 23:55:35.898056 31563 ts_tablet_manager.cc:1397] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:55:35.899396 31564 tablet_bootstrap.cc:654] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: Neither blocks nor log segments found. Creating new log.
I20260928 23:55:35.900367 31564 log.cc:824] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: Log is configured to *not* fsync() on all Append() calls
I20260928 23:55:35.901333 31562 raft_consensus.cc:348] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.901615 31562 raft_consensus.cc:374] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:35.901703 31562 raft_consensus.cc:729] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33052a024ba84cdfaf1049f7f99dc227, State: Initialized, Role: FOLLOWER
I20260928 23:55:35.902007 31563 raft_consensus.cc:348] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.902256 31562 consensus_queue.cc:260] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.902385 31563 raft_consensus.cc:374] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:35.902470 31563 raft_consensus.cc:729] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b679e7c29584663aae82273c5852232, State: Initialized, Role: FOLLOWER
I20260928 23:55:35.902787 31564 tablet_bootstrap.cc:492] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: No bootstrap required, opened a new log
I20260928 23:55:35.903049 31564 ts_tablet_manager.cc:1397] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:55:35.903013 31563 consensus_queue.cc:260] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.903505 31562 ts_tablet_manager.cc:1428] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:55:35.903772 31411 heartbeater.cc:503] Master 127.20.88.62:41401 was elected leader, sending a full tablet report...
I20260928 23:55:35.903831 31545 heartbeater.cc:503] Master 127.20.88.62:41401 was elected leader, sending a full tablet report...
I20260928 23:55:35.903982 31563 ts_tablet_manager.cc:1428] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:55:35.907013 31564 raft_consensus.cc:348] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.907332 31564 raft_consensus.cc:374] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:55:35.907410 31564 raft_consensus.cc:729] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c3aec8a04045619a956bf01e677917, State: Initialized, Role: FOLLOWER
I20260928 23:55:35.907892 31564 consensus_queue.cc:260] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.908859 31277 heartbeater.cc:503] Master 127.20.88.62:41401 was elected leader, sending a full tablet report...
I20260928 23:55:35.909010 31564 ts_tablet_manager.cc:1428] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:55:35.941907 31568 raft_consensus.cc:482] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:55:35.942153 31568 raft_consensus.cc:504] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.943400 31568 leader_election.cc:288] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c3aec8a04045619a956bf01e677917 (127.20.88.1:35695), 0b679e7c29584663aae82273c5852232 (127.20.88.2:36695)
I20260928 23:55:35.947144 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.943690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36695 (local address 127.20.88.3:59929)
0928 23:55:35.943851 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.943867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.943977 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.944322 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.944325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.944341 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:35.944462 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.944467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.945198 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.945205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.945995 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.946004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.946691 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.946695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.946805 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.946832 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.946924 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.947004 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 23:55:35.947144 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.943781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:59929 (local address 127.20.88.2:36695)
0928 23:55:35.943909 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.943916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.943932 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:55:35.944118 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.944120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.944173 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.944306 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:35.944583 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.945090 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.946126 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.946568 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.946684 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.946762 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.946940 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.947014 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:55:35.947686 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.943897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:38725 (local address 127.20.88.1:35695)
0928 23:55:35.944032 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.944037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.944362 (+ 325us) server_negotiation.cc:407] Connection header received
0928 23:55:35.944518 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.944520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.944562 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.944639 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:55:35.944903 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.945399 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.946617 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.947050 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.947189 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.947271 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.947483 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.947565 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:55:35.948037 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.943690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35695 (local address 127.20.88.3:38725)
0928 23:55:35.944235 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:35.944278 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:55:35.944401 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:55:35.944658 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:55:35.944661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:55:35.944676 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:55:35.944799 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.944805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.945509 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.945515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.946504 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:55:35.946513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.947232 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.947236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:55:35.947340 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.947370 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:55:35.947854 (+ 484us) client_negotiation.cc:240] Negotiation successful
0928 23:55:35.947926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":145,"thread_start_us":131,"threads_started":1}
I20260928 23:55:35.948065 31365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf2fd80020242fe92f2c3c5078c870f" candidate_uuid: "33052a024ba84cdfaf1049f7f99dc227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b679e7c29584663aae82273c5852232" is_pre_election: true
I20260928 23:55:35.948459 31365 raft_consensus.cc:2454] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33052a024ba84cdfaf1049f7f99dc227 in term 0.
I20260928 23:55:35.948815 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf2fd80020242fe92f2c3c5078c870f" candidate_uuid: "33052a024ba84cdfaf1049f7f99dc227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3aec8a04045619a956bf01e677917" is_pre_election: true
I20260928 23:55:35.949010 31432 leader_election.cc:302] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b679e7c29584663aae82273c5852232, 33052a024ba84cdfaf1049f7f99dc227; no voters:
I20260928 23:55:35.949138 31231 raft_consensus.cc:2454] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33052a024ba84cdfaf1049f7f99dc227 in term 0.
I20260928 23:55:35.949368 31568 raft_consensus.cc:2781] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:55:35.949484 31568 raft_consensus.cc:482] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:55:35.949546 31568 raft_consensus.cc:3037] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:35.950898 31568 raft_consensus.cc:504] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.951272 31568 leader_election.cc:288] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [CANDIDATE]: Term 1 election: Requested vote from peers d6c3aec8a04045619a956bf01e677917 (127.20.88.1:35695), 0b679e7c29584663aae82273c5852232 (127.20.88.2:36695)
I20260928 23:55:35.951640 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf2fd80020242fe92f2c3c5078c870f" candidate_uuid: "33052a024ba84cdfaf1049f7f99dc227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3aec8a04045619a956bf01e677917"
I20260928 23:55:35.951676 31365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf2fd80020242fe92f2c3c5078c870f" candidate_uuid: "33052a024ba84cdfaf1049f7f99dc227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b679e7c29584663aae82273c5852232"
I20260928 23:55:35.951792 31231 raft_consensus.cc:3037] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:35.951818 31365 raft_consensus.cc:3037] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:55:35.952999 31365 raft_consensus.cc:2454] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33052a024ba84cdfaf1049f7f99dc227 in term 1.
I20260928 23:55:35.953011 31231 raft_consensus.cc:2454] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33052a024ba84cdfaf1049f7f99dc227 in term 1.
I20260928 23:55:35.953325 31432 leader_election.cc:302] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b679e7c29584663aae82273c5852232, 33052a024ba84cdfaf1049f7f99dc227, d6c3aec8a04045619a956bf01e677917; no voters:
I20260928 23:55:35.953506 31568 raft_consensus.cc:2781] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:55:35.953713 31568 raft_consensus.cc:686] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 1 LEADER]: Becoming Leader. State: Replica: 33052a024ba84cdfaf1049f7f99dc227, State: Running, Role: LEADER
I20260928 23:55:35.954077 31568 consensus_queue.cc:237] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } }
I20260928 23:55:35.957520 31084 catalog_manager.cc:5495] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33052a024ba84cdfaf1049f7f99dc227 (127.20.88.3). New cstate: current_term: 1 leader_uuid: "33052a024ba84cdfaf1049f7f99dc227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33052a024ba84cdfaf1049f7f99dc227" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 } health_report { overall_health: UNKNOWN } } }
I20260928 23:55:35.978749 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:35.975187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.20.88.3:46775)
0928 23:55:35.975344 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:55:35.975347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:55:35.975362 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:55:35.975439 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:55:35.975441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:55:35.975498 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:55:35.975592 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:55:35.975813 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.976412 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.977316 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:55:35.977536 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:55:35.977639 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:55:35.977974 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:55:35.978414 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:55:35.978529 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:55:35.978560 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 23:55:35.978632 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:55:35.985366 31365 raft_consensus.cc:1261] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 1 FOLLOWER]: Refusing update from remote peer 33052a024ba84cdfaf1049f7f99dc227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:35.985394 31231 raft_consensus.cc:1261] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917 [term 1 FOLLOWER]: Refusing update from remote peer 33052a024ba84cdfaf1049f7f99dc227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:55:35.986181 31568 consensus_queue.cc:1035] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b679e7c29584663aae82273c5852232" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:35.986395 31572 consensus_queue.cc:1035] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c3aec8a04045619a956bf01e677917" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:55:36.004706 31084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:36.004926 31084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:55:36.007671 31084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd2ed278d7c44e24938fc19d52b3b202: Sending DeleteTablet for 3 replicas of tablet 1cf2fd80020242fe92f2c3c5078c870f
I20260928 23:55:36.008710 31211 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:33100
I20260928 23:55:36.008870 31479 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260928 23:55:36.009289 31585 tablet_replica.cc:330] T 1cf2fd80020242fe92f2c3c5078c870f P d6c3aec8a04045619a956bf01e677917: stopping tablet replica
I20260928 23:55:36.009316 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 31148
I20260928 23:55:36.009377 31345 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf2fd80020242fe92f2c3c5078c870f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:55:36 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:57612
I20260928 23:55:36.010461 31586 tablet_replica.cc:330] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: stopping tablet replica
I20260928 23:55:36.010869 31586 raft_consensus.cc:2229] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:55:36.011404 31586 raft_consensus.cc:2258] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:36.011828 31587 tablet_replica.cc:330] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: stopping tablet replica
I20260928 23:55:36.012253 31587 raft_consensus.cc:2229] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:55:36.012658 31587 raft_consensus.cc:2258] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:55:36.013458 31586 ts_tablet_manager.cc:1905] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:36.014057 31587 ts_tablet_manager.cc:1905] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:55:36.016271 31586 ts_tablet_manager.cc:1918] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:36.016450 31586 log.cc:1196] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/1cf2fd80020242fe92f2c3c5078c870f
W20260928 23:55:36.016759 31073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:35695 (error 108)
I20260928 23:55:36.016849 31587 ts_tablet_manager.cc:1918] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:55:36.016908 31586 ts_tablet_manager.cc:1939] T 1cf2fd80020242fe92f2c3c5078c870f P 33052a024ba84cdfaf1049f7f99dc227: Deleting consensus metadata
I20260928 23:55:36.016942 31587 log.cc:1196] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/1cf2fd80020242fe92f2c3c5078c870f
I20260928 23:55:36.017271 31587 ts_tablet_manager.cc:1939] T 1cf2fd80020242fe92f2c3c5078c870f P 0b679e7c29584663aae82273c5852232: Deleting consensus metadata
I20260928 23:55:36.017776 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 31282
I20260928 23:55:36.018752 31073 catalog_manager.cc:4841] TS 33052a024ba84cdfaf1049f7f99dc227 (127.20.88.3:46775): tablet 1cf2fd80020242fe92f2c3c5078c870f (table test-table [id=0d257bb8a2bc4f9cbe6b41470f030d75]) successfully deleted
I20260928 23:55:36.024398 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 31416
I20260928 23:55:36.025312 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:36.024563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35695 (local address 127.0.0.1:33112)
0928 23:55:36.025138 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:36.025215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
W20260928 23:55:36.025499 31073 catalog_manager.cc:4584] TS d6c3aec8a04045619a956bf01e677917 (127.20.88.1:35695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cf2fd80020242fe92f2c3c5078c870f: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35695: connect: Connection refused (error 111)
I20260928 23:55:36.026247 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:55:36.025947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36695 (local address 127.0.0.1:57622)
0928 23:55:36.026092 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:55:36.026162 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 23:55:36.030537 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 31054
2026-09-28T23:55:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2392 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:55:36 dist-test-slave-wvpw krb5kdc[31591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:55:36 dist-test-slave-wvpw krb5kdc[31591](info): set up 2 sockets
Sep 28 23:55:36 dist-test-slave-wvpw krb5kdc[31591](info): commencing operation
krb5kdc: starting...
W20260928 23:55:37.069842 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:55:37 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:55:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangeradmin_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangeradmin_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangerlookup_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangerlookup_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
I20260928 23:55:37.257337 20832 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHREvfx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/postgres -l logfile start
2026-09-28 23:55:40.045 UTC [31621] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:55:40.045 UTC [31621] LOG: listening on IPv4 address "127.20.88.20", port 55749
2026-09-28 23:55:40.049 UTC [31621] LOG: listening on Unix socket "/tmp/.s.PGSQL.55749"
2026-09-28 23:55:40.070 UTC [31624] LOG: database system was shut down at 2026-09-28 23:55:38 UTC
2026-09-28 23:55:40.074 UTC [31621] LOG: database system is ready to accept connections
I20260928 23:55:41.029531 20832 mini_postgres.cc:96] Postgres bound to 55749
2026-09-28 23:55:41.036 UTC [31632] FATAL: database "slave" does not exist
127.20.88.20:55749 - accepting connections
I20260928 23:55:41.036844 20832 mini_ranger.cc:161] Starting Ranger...
I20260928 23:55:41.052766 20832 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:55:41.601852 20832 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:55:41.601970 20832 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin
2026-09-28 23:55:41,900 [I] DB FLAVOR :POSTGRES
2026-09-28 23:55:41,900 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:55:41,900 [I] Checking connection..
2026-09-28 23:55:41,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:55:42,172 [I] Checking connection passed.
2026-09-28 23:55:42,173 [I] --------- Verifying version history table ---------
2026-09-28 23:55:42,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:55:42,437 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:55:42,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:55:42,688 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:55:42,688 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:55:42,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:55:42.959 UTC [31696] WARNING: there is no transaction in progress
2026-09-28 23:55:42,965 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:55:42,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:55:43,224 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:55:43,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:55:43,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:55:43,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:55:43,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvpw.c.gcp-upstream.internal','N') ;"
2026-09-28 23:55:43,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:55:44,274 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:55:44,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:55:44,551 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:55:44,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:55:44,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:55:45,070 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:55:45,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:55:46.135 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.154 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.173 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.192 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.210 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.232 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.303 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.312 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.329 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.341 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.353 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.368 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.379 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.386 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.395 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:46.404 UTC [31804] WARNING: there is no transaction in progress
2026-09-28 23:55:47,212 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:55:47,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:55:47,472 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:55:47,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:55:47,755 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:55:47,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:55:48,025 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:55:48,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:55:48,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:55:48,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvpw.c.gcp-upstream.internal';"
2026-09-28 23:55:48,773 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:55:48,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:55:49,033 [I] DB_PATCHES have already been applied
I20260928 23:55:49.039836 20832 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHREvfx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:55:49.039913 20832 mini_ranger.cc:193] Using host: 127.20.88.20
I20260928 23:55:49.043639 20832 mini_ranger.cc:227] Ranger admin URL: http://127.20.88.20:55875
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:55:50.164523 20832 mini_ranger.cc:160] Time spent starting Ranger: real 9.128s user 0.000s sys 0.012s
Sep 28, 2026 11:56:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:56:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 11:56:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:56:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:56:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:56:13.034053 20832 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangerkms_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/rangerkms_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.88.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.88.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:56:13 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:56:13.131495 20832 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:56:13.145157 20832 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:56:14.171452 20832 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:56:14.171563 20832 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms
I20260928 23:56:14.171689 20832 mini_ranger_kms.cc:227] Using postgres at 127.20.88.20:55749
2026-09-28 23:56:14,375 [I] DB FLAVOR :POSTGRES
2026-09-28 23:56:14,375 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:56:14,375 [I] Checking connection
2026-09-28 23:56:14,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:56:14,641 [I] connection success
2026-09-28 23:56:14,641 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:56:14,641 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:56:14,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:56:14,900 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:56:14,900 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:56:14,901 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:56:14,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:55749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:56:15,187 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:56:15.672495 20832 mini_ranger_kms.cc:310] Created kms service
I20260928 23:56:15.909687 20832 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:56:15.958516 20832 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:56:15 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:16.391479 20832 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:56:16.391639 20832 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:56:16.391661 20832 mini_ranger_kms.cc:240] Using host: 127.20.88.20
I20260928 23:56:16.396342 20832 mini_ranger_kms.cc:276] Ranger KMS PID: 31985
I20260928 23:56:16.396510 20832 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.88.20:56651
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:56:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:56:18.621124 20832 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.490s user 0.017s sys 0.172s
I20260928 23:56:18.621248 20832 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:56:18.621344 20832 mini_ranger_kms.cc:401] 127.20.88.20:56651/kms/v1/keys
Sep 28 23:56:18 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:56:19 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
Sep 28 23:56:19 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
Sep 28 23:56:21 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:56:21.828310 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36361
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:36361
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:56651/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:56:21.945382 32035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:56:21.945756 32035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:56:21.945837 32035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:56:21.950169 32035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:56:21.950275 32035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:56:21.950312 32035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:56:21.950376 32035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:56:21.950416 32035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:56:21.956365 32035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.88.20:56651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:36361
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:56:21.957624 32035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:56:21.958703 32035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:56:21.968400 32035 server_base.cc:1034] running on GCE node
W20260928 23:56:21.968349 32040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:21.968349 32043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:21.968375 32041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:21.969388 32035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:56:21.970620 32035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:56:21.971858 32035 hybrid_clock.cc:648] HybridClock initialized: now 1790639781971836 us; error 35 us; skew 500 ppm
Sep 28 23:56:21 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639781, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:56:21.979079 32035 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:56:21.980291 32035 webserver.cc:460] Webserver started at http://127.20.88.62:36271/ using document root <none> and password file <none>
I20260928 23:56:21.980935 32035 fs_manager.cc:359] Metadata directory not provided
I20260928 23:56:21.981027 32035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:56:21.981313 32035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:56:21 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639781, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:22.037933 32035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "99f661e4d8794a0dbb61b0da0d842dcd"
format_stamp: "Formatted at 2026-09-28 23:56:22 on dist-test-slave-wvpw"
server_key: "655467a6189bb0df43b969bbc9367768"
server_key_iv: "d581289b412dc3a065b8d7402b8de695"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:22.038636 32035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "99f661e4d8794a0dbb61b0da0d842dcd"
format_stamp: "Formatted at 2026-09-28 23:56:22 on dist-test-slave-wvpw"
server_key: "655467a6189bb0df43b969bbc9367768"
server_key_iv: "d581289b412dc3a065b8d7402b8de695"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:22.043012 32035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:56:22.058347 32050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:22.059790 32035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:56:22.059937 32035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "99f661e4d8794a0dbb61b0da0d842dcd"
format_stamp: "Formatted at 2026-09-28 23:56:22 on dist-test-slave-wvpw"
server_key: "655467a6189bb0df43b969bbc9367768"
server_key_iv: "d581289b412dc3a065b8d7402b8de695"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:22.060091 32035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:56:22.082515 32035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:56:22.086212 32035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:56:22.086475 32035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:56:22.094229 32035 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:36361
I20260928 23:56:22.094301 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:36361 every 8 connection(s)
I20260928 23:56:22.095361 32035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:56:22.097010 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 32035
I20260928 23:56:22.097185 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:56:22.097500 20832 external_mini_cluster.cc:1428] Setting key 4f7e4d8c32b19af569934391e31c5d42
I20260928 23:56:22.099156 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:56:22.106132 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: Bootstrap starting.
Sep 28 23:56:22 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:56:22.108698 32103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: Neither blocks nor log segments found. Creating new log.
I20260928 23:56:22.109779 32103 log.cc:824] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:56:22.111776 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: No bootstrap required, opened a new log
I20260928 23:56:22.112915 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:22.099058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.20.88.62:36361)
0928 23:56:22.099548 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:22.099578 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:22.099616 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:56:22.100375 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:22.100394 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:22.100739 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:22.101039 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:56:22.102005 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:22.103080 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:22.104330 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:22.104829 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:22.104962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:22.108091 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:56:22.108097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:56:22.108111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:56:22.108143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:56:22.110019 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:22.110480 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:22.110488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:22.110499 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:22.110602 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:22.110905 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:22.110908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:22.110910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:22.111325 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:56:22.111539 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:22.111832 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 23:56:22.112120 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260928 23:56:22.115396 32103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } }
I20260928 23:56:22.115684 32103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:56:22.115780 32103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f661e4d8794a0dbb61b0da0d842dcd, State: Initialized, Role: FOLLOWER
I20260928 23:56:22.116220 32103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } }
I20260928 23:56:22.116459 32103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:56:22.116547 32103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:56:22.116704 32103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:56:22.118064 32103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } }
I20260928 23:56:22.118448 32103 leader_election.cc:302] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f661e4d8794a0dbb61b0da0d842dcd; no voters:
I20260928 23:56:22.119001 32103 leader_election.cc:288] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:56:22.119454 32109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:56:22.119735 32109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [term 1 LEADER]: Becoming Leader. State: Replica: 99f661e4d8794a0dbb61b0da0d842dcd, State: Running, Role: LEADER
I20260928 23:56:22.120105 32109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } }
I20260928 23:56:22.120611 32103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:56:22.121465 32109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f661e4d8794a0dbb61b0da0d842dcd. Latest consensus state: current_term: 1 leader_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } } }
I20260928 23:56:22.121711 32109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [sys.catalog]: This master's current role is: LEADER
I20260928 23:56:22.122139 32117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:56:22.123003 32110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f661e4d8794a0dbb61b0da0d842dcd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36361 } } }
I20260928 23:56:22.123246 32110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd [sys.catalog]: This master's current role is: LEADER
I20260928 23:56:22.126588 32117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:56:22.127538 32103 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:56:22.137948 32117 catalog_manager.cc:1347] Generated new cluster ID: 760bc65546de4b799a00cbae0fc14dc5
I20260928 23:56:22.138048 32117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:56:22.180155 32117 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:56:22.181543 32117 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:56:22.186299 32117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: Generated new TSK 0
I20260928 23:56:22.187148 32117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:56:22.880669 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:56651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:56:23.046319 32148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:56:23.046677 32148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:56:23.046787 32148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:56:23.051451 32148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:56:23.051568 32148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:56:23.051692 32148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:56:23.059939 32148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.88.20:56651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:56:23.061475 32148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:56:23.062909 32148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:56:23.076735 32156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:23.076750 32158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:23.077554 32148 server_base.cc:1034] running on GCE node
W20260928 23:56:23.076803 32155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:23.078615 32148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:56:23.079521 32148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:56:23.080726 32148 hybrid_clock.cc:648] HybridClock initialized: now 1790639783080694 us; error 56 us; skew 500 ppm
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:56:23.084005 32148 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:56:23.085216 32148 webserver.cc:460] Webserver started at http://127.20.88.1:46805/ using document root <none> and password file <none>
I20260928 23:56:23.085858 32148 fs_manager.cc:359] Metadata directory not provided
I20260928 23:56:23.085960 32148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:56:23.086192 32148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:23.117118 32148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "3e6c7bdd6a0148bc9795504595730646"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "41628f3c60a97fabaca1f23fc308d465"
server_key_iv: "ad938cd59e517f8a14e2ca274d1a7451"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.118007 32148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "3e6c7bdd6a0148bc9795504595730646"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "41628f3c60a97fabaca1f23fc308d465"
server_key_iv: "ad938cd59e517f8a14e2ca274d1a7451"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.126087 32148 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260928 23:56:23.156893 32165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.162958 32148 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260928 23:56:23.163159 32148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "3e6c7bdd6a0148bc9795504595730646"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "41628f3c60a97fabaca1f23fc308d465"
server_key_iv: "ad938cd59e517f8a14e2ca274d1a7451"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.163354 32148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:56:23.182751 32148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:56:23.186794 32148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:56:23.187002 32148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:56:23.187760 32148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:56:23.188887 32148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:56:23.188954 32148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.189008 32148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:56:23.189050 32148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.199904 32148 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:41169
I20260928 23:56:23.200066 32281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:41169 every 8 connection(s)
I20260928 23:56:23.201182 32148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:56:23.205449 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 32148
I20260928 23:56:23.205617 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:56:23.205947 20832 external_mini_cluster.cc:1428] Setting key 6b48a5164a835581868bd815e922fe4f
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:56:23.221328 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.203506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:59835 (local address 127.20.88.62:36361)
0928 23:56:23.207049 (+ 3543us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.207054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.207071 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:56:23.207113 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.207116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.207163 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.207253 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:56:23.208900 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.209790 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.211639 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.212004 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.212134 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.215400 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:56:23.215405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:56:23.215407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:56:23.215441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:56:23.217593 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.218306 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.218310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.218312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.218409 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.218834 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.218837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.218839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.219160 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:56:23.219259 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.220984 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.221138 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3439,"thread_start_us":102,"threads_started":1}
I20260928 23:56:23.221421 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.204035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36361 (local address 127.20.88.1:59835)
0928 23:56:23.205367 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.205428 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.206575 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.207590 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.207601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.207965 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:56:23.208705 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.208725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.209988 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.209993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.211474 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.211486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.212387 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.212392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.212622 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.213362 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:56:23.213385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:56:23.215157 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:56:23.217754 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.217761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.217785 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.218178 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.218513 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.218516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.218518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.218717 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.219317 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:56:23.219324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:56:23.219706 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.219913 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.220310 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":203,"threads_started":1}
I20260928 23:56:23.223073 32282 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36361
I20260928 23:56:23.223363 32282 heartbeater.cc:464] Registering TS with master...
I20260928 23:56:23.224030 32282 heartbeater.cc:511] Master 127.20.88.62:36361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:56:23.225854 32067 ts_manager.cc:194] Registered new tserver with Master: 3e6c7bdd6a0148bc9795504595730646 (127.20.88.1:41169)
I20260928 23:56:23.227388 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:59835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:56:23.248490 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:56651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:56:23.392973 32288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:56:23.393386 32288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:56:23.393482 32288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:56:23.398231 32288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:56:23.398325 32288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:56:23.398438 32288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:56:23.404573 32288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.88.20:56651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:56:23.405791 32288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:56:23.406976 32288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:56:23.418885 32288 server_base.cc:1034] running on GCE node
W20260928 23:56:23.419067 32297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:23.419106 32295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:23.419358 32294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:23.419878 32288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:56:23.420712 32288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:56:23.421926 32288 hybrid_clock.cc:648] HybridClock initialized: now 1790639783421848 us; error 96 us; skew 500 ppm
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:56:23.425918 32288 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:56:23.427250 32288 webserver.cc:460] Webserver started at http://127.20.88.2:43409/ using document root <none> and password file <none>
I20260928 23:56:23.428054 32288 fs_manager.cc:359] Metadata directory not provided
I20260928 23:56:23.428318 32288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:56:23.428834 32288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:23.444983 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "8882d2078a17414bb56b41367fa34d3e"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "c143e98c4a9d6d449a3e42383e96aeae"
server_key_iv: "ffe4fba4a7565e88c102cb8e2cd7c59c"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.446069 32288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "8882d2078a17414bb56b41367fa34d3e"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "c143e98c4a9d6d449a3e42383e96aeae"
server_key_iv: "ffe4fba4a7565e88c102cb8e2cd7c59c"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.451885 32288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 23:56:23.468926 32304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.470741 32288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:56:23.470898 32288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "8882d2078a17414bb56b41367fa34d3e"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "c143e98c4a9d6d449a3e42383e96aeae"
server_key_iv: "ffe4fba4a7565e88c102cb8e2cd7c59c"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.471051 32288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:56:23.490981 32288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:56:23.494963 32288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:56:23.495174 32288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:56:23.495939 32288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:56:23.497062 32288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:56:23.497133 32288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.497192 32288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:56:23.497222 32288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.511345 32288 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:40757
I20260928 23:56:23.511379 32417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:40757 every 8 connection(s)
I20260928 23:56:23.512878 32288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:56:23.518167 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 32288
I20260928 23:56:23.518332 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:56:23.518659 20832 external_mini_cluster.cc:1428] Setting key eb69c3a660b7476eb014681214bc8484
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:56:23.534126 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.517197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:32933 (local address 127.20.88.62:36361)
0928 23:56:23.517345 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.517350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.518277 (+ 927us) server_negotiation.cc:407] Connection header received
0928 23:56:23.519962 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.519966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.520024 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.520084 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:56:23.521945 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.522754 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.524590 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.524965 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.525098 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.528426 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:56:23.528431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:56:23.528433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:56:23.528470 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:56:23.530446 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.531125 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.531129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.531130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.531213 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.531624 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.531627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.531629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.531838 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:56:23.531958 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.533786 (+ 1828us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.533950 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:56:23.534442 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.515888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36361 (local address 127.20.88.2:32933)
0928 23:56:23.518074 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.518144 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.519679 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.520565 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.520578 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.520979 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:56:23.521738 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.521760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.522939 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.522942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.524433 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.524447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.525309 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.525314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.525534 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.526281 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:56:23.526304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:56:23.528088 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:56:23.530620 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.530628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.530650 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.530979 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.531318 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.531322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.531324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.531499 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.531959 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:56:23.531970 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:56:23.532391 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.532631 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.533024 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1936,"thread_start_us":157,"threads_started":1}
I20260928 23:56:23.535982 32418 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36361
I20260928 23:56:23.536461 32418 heartbeater.cc:464] Registering TS with master...
I20260928 23:56:23.537201 32418 heartbeater.cc:511] Master 127.20.88.62:36361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:56:23.538483 32067 ts_manager.cc:194] Registered new tserver with Master: 8882d2078a17414bb56b41367fa34d3e (127.20.88.2:40757)
I20260928 23:56:23.539288 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:32933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:56:23.556716 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:56651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
W20260928 23:56:23.721542 32423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:56:23.721948 32423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:56:23.722013 32423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:56:23.728648 32423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:56:23.728739 32423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:56:23.728827 32423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:56:23.736915 32423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.88.20:56651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:56:23.738592 32423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:56:23.740026 32423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:56:23.753362 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:23.753774 32423 server_base.cc:1034] running on GCE node
W20260928 23:56:23.754087 32429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:56:23.754251 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:56:23.756072 32423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:56:23.757030 32423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:56:23.760301 32423 hybrid_clock.cc:648] HybridClock initialized: now 1790639783760034 us; error 274 us; skew 500 ppm
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:56:23.764125 32423 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:56:23.765713 32423 webserver.cc:460] Webserver started at http://127.20.88.3:41873/ using document root <none> and password file <none>
I20260928 23:56:23.776044 32423 fs_manager.cc:359] Metadata directory not provided
I20260928 23:56:23.776346 32423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:56:23.776715 32423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:23.803140 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "aae4b8dadede42118e43455fe1dc2a78"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "85c5648604184291d37f13d8fc7cad62"
server_key_iv: "74943ee6ab278892929b32aab9bb839d"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.803925 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "aae4b8dadede42118e43455fe1dc2a78"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "85c5648604184291d37f13d8fc7cad62"
server_key_iv: "74943ee6ab278892929b32aab9bb839d"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.820601 32423 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260928 23:56:23.837441 32438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.839293 32423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:56:23.839449 32423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "aae4b8dadede42118e43455fe1dc2a78"
format_stamp: "Formatted at 2026-09-28 23:56:23 on dist-test-slave-wvpw"
server_key: "85c5648604184291d37f13d8fc7cad62"
server_key_iv: "74943ee6ab278892929b32aab9bb839d"
server_key_version: "kuduclusterkey@0"
I20260928 23:56:23.839605 32423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:56:23.871556 32423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:56:23.875511 32423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:56:23.875775 32423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:56:23.876709 32423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:56:23.877940 32423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:56:23.878019 32423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.878078 32423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:56:23.878109 32423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:56:23.889701 32423 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:33575
I20260928 23:56:23.889861 32551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:33575 every 8 connection(s)
I20260928 23:56:23.890944 32423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:56:23.897241 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 32423
I20260928 23:56:23.897382 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:56:23.897657 20832 external_mini_cluster.cc:1428] Setting key afef4eac2e3268bbf95539f2d6568748
Sep 28 23:56:23 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:56:23.908180 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.893123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:44439 (local address 127.20.88.62:36361)
0928 23:56:23.893279 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.893284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.894445 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 23:56:23.895798 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.895802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.895861 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.895956 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:56:23.897629 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.898475 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.900273 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.900635 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.900741 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.903777 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:56:23.903782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:56:23.903785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:56:23.903820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:56:23.905615 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.906286 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.906290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.906293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.906386 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.906804 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.906808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.906810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.907063 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:56:23.907184 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.907848 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.907989 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:56:23.909241 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.893607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:36361 (local address 127.20.88.3:44439)
0928 23:56:23.894255 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.894320 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.895513 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.896198 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.896272 (+ 74us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.896663 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:56:23.897419 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.897444 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.898651 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.898654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.900088 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.900101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.900806 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.900811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.901028 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.901806 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:56:23.901827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:56:23.903568 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:56:23.905755 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.905761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.905781 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.906170 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.906488 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:56:23.906491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:56:23.906493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:56:23.906693 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:56:23.907191 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:56:23.907199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:56:23.907553 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.907773 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.908131 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":167,"threads_started":1}
I20260928 23:56:23.910561 32552 heartbeater.cc:347] Connected to a master server at 127.20.88.62:36361
I20260928 23:56:23.910830 32552 heartbeater.cc:464] Registering TS with master...
I20260928 23:56:23.911417 32552 heartbeater.cc:511] Master 127.20.88.62:36361 requested a full tablet report, sending...
I20260928 23:56:23.912642 32067 ts_manager.cc:194] Registered new tserver with Master: aae4b8dadede42118e43455fe1dc2a78 (127.20.88.3:33575)
I20260928 23:56:23.913431 32067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:44439
I20260928 23:56:23.914256 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:56:23.925006 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.916845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.20.88.62:36361)
0928 23:56:23.916984 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.916992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.917203 (+ 211us) server_negotiation.cc:407] Connection header received
0928 23:56:23.917361 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.917366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.917421 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.917517 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:56:23.918235 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.918777 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.919639 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.919859 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.919970 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.921528 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:56:23.921532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:56:23.921541 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:56:23.921566 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:56:23.923238 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.923671 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.923676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.923678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.923761 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:56:23.924048 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:56:23.924051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:56:23.924053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:56:23.924311 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:56:23.924447 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.924694 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.924814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:56:23.928474 32066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:56:23.931722 32066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:56:23.952690 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.941970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.20.88.1:41169)
0928 23:56:23.942369 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.942376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.943966 (+ 1590us) server_negotiation.cc:407] Connection header received
0928 23:56:23.944312 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.944318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.944581 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.944809 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:56:23.945372 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.947533 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.950737 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.951645 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.951728 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.951899 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.952402 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.952512 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":100,"threads_started":1}
I20260928 23:56:23.954185 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.941853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:41169 (local address 127.0.0.1:52778)
0928 23:56:23.943810 (+ 1957us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.943871 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.944087 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.945056 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.945063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.945087 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:56:23.945235 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.945242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.947693 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.947696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.950572 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.950581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.951992 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.951995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.952141 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.952187 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.953871 (+ 1684us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.954041 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1770,"thread_start_us":128,"threads_started":1}
I20260928 23:56:23.958036 32203 tablet_service.cc:1467] Processing CreateTablet for tablet 7df74bca95fa405ea8fb420955a55d67 (DEFAULT_TABLE table=test-table [id=d735ad27815942adbd7424ae2000e256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:56:23.960080 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.943162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40757 (local address 127.0.0.1:55064)
0928 23:56:23.948674 (+ 5512us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.948689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.948776 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.949919 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.949923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.949939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:56:23.950045 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.950051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.954005 (+ 3954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.954009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.955812 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.955823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.959629 (+ 3806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.959632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.959739 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.959764 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.959838 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.959914 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"thread_start_us":85,"threads_started":1}
I20260928 23:56:23.960386 32203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df74bca95fa405ea8fb420955a55d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:56:23.960958 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.947559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:33575 (local address 127.0.0.1:43928)
0928 23:56:23.950816 (+ 3257us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:23.950833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:23.950917 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:23.951634 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:23.951637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:23.951655 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:56:23.951773 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.951779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.954245 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.954250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.957091 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:23.957103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.960582 (+ 3479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.960586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:23.960684 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.960706 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:56:23.960774 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:56:23.960841 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3191,"thread_start_us":94,"threads_started":1}
I20260928 23:56:23.961500 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.947962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.20.88.3:33575)
0928 23:56:23.948994 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.949006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.951015 (+ 2009us) server_negotiation.cc:407] Connection header received
0928 23:56:23.951098 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.951115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.951335 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.951487 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:56:23.951894 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.953161 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.957271 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.958108 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.958215 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.958359 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.961240 (+ 2881us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.961355 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":156,"threads_started":1}
I20260928 23:56:23.965119 32468 tablet_service.cc:1467] Processing CreateTablet for tablet 7df74bca95fa405ea8fb420955a55d67 (DEFAULT_TABLE table=test-table [id=d735ad27815942adbd7424ae2000e256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:56:23.966497 32468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df74bca95fa405ea8fb420955a55d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:56:23.973098 32570 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: Bootstrap starting.
I20260928 23:56:23.976027 32570 tablet_bootstrap.cc:654] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: Neither blocks nor log segments found. Creating new log.
I20260928 23:56:23.976715 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:23.946761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.20.88.2:40757)
0928 23:56:23.949222 (+ 2461us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:23.949238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:23.949263 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:56:23.949349 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:23.949355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:23.949590 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:23.949763 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:56:23.950184 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.953876 (+ 3692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.958467 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:23.959285 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:23.959400 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:23.959563 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:23.976371 (+ 16808us) server_negotiation.cc:299] Negotiation successful
0928 23:56:23.976556 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331,"thread_start_us":149,"threads_started":1}
I20260928 23:56:23.977260 32570 log.cc:824] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: Log is configured to *not* fsync() on all Append() calls
I20260928 23:56:23.979027 32571 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Bootstrap starting.
I20260928 23:56:23.982208 32571 tablet_bootstrap.cc:654] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Neither blocks nor log segments found. Creating new log.
I20260928 23:56:23.981765 32352 tablet_service.cc:1467] Processing CreateTablet for tablet 7df74bca95fa405ea8fb420955a55d67 (DEFAULT_TABLE table=test-table [id=d735ad27815942adbd7424ae2000e256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:56:23.983148 32352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df74bca95fa405ea8fb420955a55d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:56:23.983233 32571 log.cc:824] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Log is configured to *not* fsync() on all Append() calls
I20260928 23:56:23.986091 32571 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: No bootstrap required, opened a new log
I20260928 23:56:23.986392 32571 ts_tablet_manager.cc:1397] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:56:23.991010 32571 raft_consensus.cc:348] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:23.991583 32571 raft_consensus.cc:374] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:56:23.991714 32571 raft_consensus.cc:729] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae4b8dadede42118e43455fe1dc2a78, State: Initialized, Role: FOLLOWER
I20260928 23:56:23.992403 32571 consensus_queue.cc:260] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:23.993084 32570 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: No bootstrap required, opened a new log
I20260928 23:56:23.993705 32571 ts_tablet_manager.cc:1428] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:56:23.994123 32552 heartbeater.cc:503] Master 127.20.88.62:36361 was elected leader, sending a full tablet report...
I20260928 23:56:23.995891 32570 ts_tablet_manager.cc:1397] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260928 23:56:24.000815 32575 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Bootstrap starting.
I20260928 23:56:24.001206 32570 raft_consensus.cc:348] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.001586 32570 raft_consensus.cc:374] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:56:24.001691 32570 raft_consensus.cc:729] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6c7bdd6a0148bc9795504595730646, State: Initialized, Role: FOLLOWER
I20260928 23:56:24.002321 32570 consensus_queue.cc:260] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.003482 32575 tablet_bootstrap.cc:654] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Neither blocks nor log segments found. Creating new log.
I20260928 23:56:24.003546 32570 ts_tablet_manager.cc:1428] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:56:24.004995 32575 log.cc:824] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:56:24.005411 32282 heartbeater.cc:503] Master 127.20.88.62:36361 was elected leader, sending a full tablet report...
I20260928 23:56:24.010423 32575 tablet_bootstrap.cc:492] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: No bootstrap required, opened a new log
I20260928 23:56:24.010758 32575 ts_tablet_manager.cc:1397] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 23:56:24.014925 32575 raft_consensus.cc:348] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.015408 32575 raft_consensus.cc:374] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:56:24.015509 32575 raft_consensus.cc:729] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8882d2078a17414bb56b41367fa34d3e, State: Initialized, Role: FOLLOWER
I20260928 23:56:24.016296 32575 consensus_queue.cc:260] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.021648 32575 ts_tablet_manager.cc:1428] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260928 23:56:24.022082 32418 heartbeater.cc:503] Master 127.20.88.62:36361 was elected leader, sending a full tablet report...
Sep 28 23:56:24 dist-test-slave-wvpw krb5kdc[31591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639783, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:56:24.095695 32576 raft_consensus.cc:482] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:56:24.095945 32576 raft_consensus.cc:504] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.097468 32576 leader_election.cc:288] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8882d2078a17414bb56b41367fa34d3e (127.20.88.2:40757), aae4b8dadede42118e43455fe1dc2a78 (127.20.88.3:33575)
I20260928 23:56:24.102972 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.098410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:36423 (local address 127.20.88.2:40757)
0928 23:56:24.098546 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:24.098551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:24.098565 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:56:24.098624 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:24.098626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:24.098676 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:24.098749 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:56:24.099165 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.099967 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.101600 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.102220 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.102301 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:24.102370 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:24.102754 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 23:56:24.102834 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:56:24.103634 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.097711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40757 (local address 127.20.88.1:36423)
0928 23:56:24.098177 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:24.098193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:24.098302 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:24.098866 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:24.098869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:24.098890 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:56:24.099034 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:24.099041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.100121 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.100124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:24.101446 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:24.101457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.102462 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.102465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:24.102616 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:24.102638 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:56:24.103394 (+ 756us) client_negotiation.cc:240] Negotiation successful
0928 23:56:24.103477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":55,"thread_start_us":106,"threads_started":1}
I20260928 23:56:24.104305 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.098539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:53281 (local address 127.20.88.3:33575)
0928 23:56:24.098671 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:24.098676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:24.099925 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 23:56:24.100109 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:24.100112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:24.100321 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:24.100387 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:56:24.100825 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.101560 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.103014 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.103592 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.103669 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:24.103731 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:24.104053 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 23:56:24.104172 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:56:24.104485 32372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df74bca95fa405ea8fb420955a55d67" candidate_uuid: "3e6c7bdd6a0148bc9795504595730646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8882d2078a17414bb56b41367fa34d3e" is_pre_election: true
I20260928 23:56:24.104709 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.097998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:33575 (local address 127.20.88.1:53281)
0928 23:56:24.099834 (+ 1836us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:24.099850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:56:24.100001 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:56:24.100526 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:56:24.100529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:56:24.100547 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:56:24.100697 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:24.100704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.101722 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.101725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:24.102876 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:56:24.102887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.103823 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.103826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:56:24.103924 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:24.103943 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:56:24.104493 (+ 550us) client_negotiation.cc:240] Negotiation successful
0928 23:56:24.104563 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1765,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260928 23:56:24.104877 32372 raft_consensus.cc:2454] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6c7bdd6a0148bc9795504595730646 in term 0.
I20260928 23:56:24.105255 32506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df74bca95fa405ea8fb420955a55d67" candidate_uuid: "3e6c7bdd6a0148bc9795504595730646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae4b8dadede42118e43455fe1dc2a78" is_pre_election: true
I20260928 23:56:24.105432 32168 leader_election.cc:302] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6c7bdd6a0148bc9795504595730646, 8882d2078a17414bb56b41367fa34d3e; no voters:
I20260928 23:56:24.105533 32506 raft_consensus.cc:2454] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6c7bdd6a0148bc9795504595730646 in term 0.
I20260928 23:56:24.105778 32576 raft_consensus.cc:2781] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:56:24.105863 32576 raft_consensus.cc:482] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:56:24.105901 32576 raft_consensus.cc:3037] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:56:24.107410 32576 raft_consensus.cc:504] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.107865 32576 leader_election.cc:288] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [CANDIDATE]: Term 1 election: Requested vote from peers 8882d2078a17414bb56b41367fa34d3e (127.20.88.2:40757), aae4b8dadede42118e43455fe1dc2a78 (127.20.88.3:33575)
I20260928 23:56:24.108345 32372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df74bca95fa405ea8fb420955a55d67" candidate_uuid: "3e6c7bdd6a0148bc9795504595730646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8882d2078a17414bb56b41367fa34d3e"
I20260928 23:56:24.108487 32372 raft_consensus.cc:3037] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:56:24.108543 32506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df74bca95fa405ea8fb420955a55d67" candidate_uuid: "3e6c7bdd6a0148bc9795504595730646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae4b8dadede42118e43455fe1dc2a78"
I20260928 23:56:24.108664 32506 raft_consensus.cc:3037] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:56:24.109915 32372 raft_consensus.cc:2454] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6c7bdd6a0148bc9795504595730646 in term 1.
I20260928 23:56:24.110002 32506 raft_consensus.cc:2454] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6c7bdd6a0148bc9795504595730646 in term 1.
I20260928 23:56:24.110484 32171 leader_election.cc:302] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6c7bdd6a0148bc9795504595730646, aae4b8dadede42118e43455fe1dc2a78; no voters:
I20260928 23:56:24.110725 32576 raft_consensus.cc:2781] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:56:24.110927 32576 raft_consensus.cc:686] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 1 LEADER]: Becoming Leader. State: Replica: 3e6c7bdd6a0148bc9795504595730646, State: Running, Role: LEADER
I20260928 23:56:24.111299 32576 consensus_queue.cc:237] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } }
I20260928 23:56:24.115109 32066 catalog_manager.cc:5495] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e6c7bdd6a0148bc9795504595730646 (127.20.88.1). New cstate: current_term: 1 leader_uuid: "3e6c7bdd6a0148bc9795504595730646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c7bdd6a0148bc9795504595730646" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 41169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 } health_report { overall_health: UNKNOWN } } }
W20260928 23:56:24.145232 32553 tablet.cc:2369] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:56:24.148032 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.142541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.20.88.1:41169)
0928 23:56:24.142699 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:56:24.142704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:56:24.142720 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:56:24.142815 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:56:24.142818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:56:24.142889 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:56:24.143015 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:56:24.143379 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.144194 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.145594 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:56:24.145931 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:56:24.146041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:56:24.146506 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:56:24.147122 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:56:24.147247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:56:24.147802 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 23:56:24.147879 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:56:24.155432 32506 raft_consensus.cc:1261] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 1 FOLLOWER]: Refusing update from remote peer 3e6c7bdd6a0148bc9795504595730646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:56:24.156325 32583 consensus_queue.cc:1035] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae4b8dadede42118e43455fe1dc2a78" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:56:24.156416 32372 raft_consensus.cc:1261] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 1 FOLLOWER]: Refusing update from remote peer 3e6c7bdd6a0148bc9795504595730646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:56:24.157020 32583 consensus_queue.cc:1035] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8882d2078a17414bb56b41367fa34d3e" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:56:24.173930 32586 mvcc.cc:205] Tried to move back new op lower bound from 7334460555891159040 to 7334460555722870784. Current Snapshot: MvccSnapshot[applied={T|T < 7334460555891159040}]
I20260928 23:56:24.190687 32589 mvcc.cc:205] Tried to move back new op lower bound from 7334460555891159040 to 7334460555722870784. Current Snapshot: MvccSnapshot[applied={T|T < 7334460555891159040}]
I20260928 23:56:24.191876 32067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:56:24.192047 32067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:56:24.194957 32067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99f661e4d8794a0dbb61b0da0d842dcd: Sending DeleteTablet for 3 replicas of tablet 7df74bca95fa405ea8fb420955a55d67
I20260928 23:56:24.196298 32203 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df74bca95fa405ea8fb420955a55d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:56:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:52778
I20260928 23:56:24.196872 32468 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df74bca95fa405ea8fb420955a55d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:56:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:43928
I20260928 23:56:24.197885 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df74bca95fa405ea8fb420955a55d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:56:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260928 23:56:24.198173 32595 tablet_replica.cc:330] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646: stopping tablet replica
I20260928 23:56:24.198504 32595 raft_consensus.cc:2229] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:56:24.198868 32595 raft_consensus.cc:2258] T 7df74bca95fa405ea8fb420955a55d67 P 3e6c7bdd6a0148bc9795504595730646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:56:24.199170 32596 tablet_replica.cc:330] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: stopping tablet replica
I20260928 23:56:24.199437 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 32148
I20260928 23:56:24.199514 32596 raft_consensus.cc:2229] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:56:24.199818 32596 raft_consensus.cc:2258] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:56:24.207464 32596 ts_tablet_manager.cc:1905] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:56:24.207870 32597 tablet_replica.cc:330] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: stopping tablet replica
I20260928 23:56:24.208401 32597 raft_consensus.cc:2229] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:56:24.208982 32597 raft_consensus.cc:2258] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:56:24.211429 32597 ts_tablet_manager.cc:1905] T 7df74bca95fa405ea8fb420955a55d67 P 8882d2078a17414bb56b41367fa34d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:56:24.211869 32054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:41169 (error 108)
I20260928 23:56:24.212430 32596 ts_tablet_manager.cc:1918] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:56:24.212654 32596 log.cc:1196] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/7df74bca95fa405ea8fb420955a55d67
I20260928 23:56:24.213101 32596 ts_tablet_manager.cc:1939] T 7df74bca95fa405ea8fb420955a55d67 P aae4b8dadede42118e43455fe1dc2a78: Deleting consensus metadata
I20260928 23:56:24.214610 32054 catalog_manager.cc:4841] TS aae4b8dadede42118e43455fe1dc2a78 (127.20.88.3:33575): tablet 7df74bca95fa405ea8fb420955a55d67 (table test-table [id=d735ad27815942adbd7424ae2000e256]) successfully deleted
I20260928 23:56:24.214938 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 32288
I20260928 23:56:24.217486 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.217107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:41169 (local address 127.0.0.1:52802)
0928 23:56:24.217320 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:24.217394 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:41169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260928 23:56:24.217821 32054 catalog_manager.cc:4584] TS 3e6c7bdd6a0148bc9795504595730646 (127.20.88.1:41169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7df74bca95fa405ea8fb420955a55d67: Network error: Client connection negotiation failed: client connection to 127.20.88.1:41169: connect: Connection refused (error 111)
I20260928 23:56:24.224012 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 32423
I20260928 23:56:24.227770 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:56:24.224197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:40757 (local address 127.0.0.1:55068)
0928 23:56:24.227584 (+ 3387us) negotiation.cc:107] Waiting for socket to connect
0928 23:56:24.227660 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3270}
I20260928 23:56:24.234338 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 32035
I20260928 23:56:24.241408 20832 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:56:24.587463 20832 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:56:24.587648 20832 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:56:24.933579 20832 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:56:24.933 UTC [31621] LOG: received smart shutdown request
2026-09-28 23:56:24.937 UTC [31621] LOG: background worker "logical replication launcher" (PID 31630) exited with exit code 1
2026-09-28 23:56:24.938 UTC [31625] LOG: shutting down
2026-09-28 23:56:25.143 UTC [31621] LOG: database system is shut down
2026-09-28T23:56:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49229 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:56:25 dist-test-slave-wvpw krb5kdc[32607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:56:25 dist-test-slave-wvpw krb5kdc[32607](info): set up 2 sockets
Sep 28 23:56:25 dist-test-slave-wvpw krb5kdc[32607](info): commencing operation
krb5kdc: starting...
W20260928 23:56:26.316926 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:56:26 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:56:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangeradmin_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangeradmin_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangerlookup_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangerlookup_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
I20260928 23:56:26.508666 20832 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHREvfx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/postgres -l logfile start
2026-09-28 23:56:29.298 UTC [32637] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:56:29.298 UTC [32637] LOG: listening on IPv4 address "127.20.88.20", port 45189
2026-09-28 23:56:29.303 UTC [32637] LOG: listening on Unix socket "/tmp/.s.PGSQL.45189"
2026-09-28 23:56:29.326 UTC [32640] LOG: database system was shut down at 2026-09-28 23:56:27 UTC
2026-09-28 23:56:29.331 UTC [32637] LOG: database system is ready to accept connections
I20260928 23:56:30.284860 20832 mini_postgres.cc:96] Postgres bound to 45189
2026-09-28 23:56:30.291 UTC [32648] FATAL: database "slave" does not exist
127.20.88.20:45189 - accepting connections
I20260928 23:56:30.292433 20832 mini_ranger.cc:161] Starting Ranger...
I20260928 23:56:30.308735 20832 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:56:30.911351 20832 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:56:30.911429 20832 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin
2026-09-28 23:56:31,191 [I] DB FLAVOR :POSTGRES
2026-09-28 23:56:31,191 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:56:31,191 [I] Checking connection..
2026-09-28 23:56:31,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:56:31,436 [I] Checking connection passed.
2026-09-28 23:56:31,436 [I] --------- Verifying version history table ---------
2026-09-28 23:56:31,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:56:31,707 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:56:31,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:56:31,959 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:56:31,959 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:56:31,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:56:32.215 UTC [32712] WARNING: there is no transaction in progress
2026-09-28 23:56:32,222 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:56:32,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:56:32,482 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:56:32,482 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:56:32,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:56:32,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:56:32,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvpw.c.gcp-upstream.internal','N') ;"
2026-09-28 23:56:33,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:56:33,479 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:56:33,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:56:33,727 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:56:33,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:56:33,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:56:34,228 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:56:34,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:56:35.342 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.363 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.383 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.404 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.425 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.451 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.521 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.530 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.548 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.562 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.575 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.594 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.606 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.614 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.623 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:35.633 UTC [352] WARNING: there is no transaction in progress
2026-09-28 23:56:36,482 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:56:36,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:56:36,732 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:56:36,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:56:36,995 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:56:36,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:56:37,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:56:37,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:56:37,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:56:37,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvpw.c.gcp-upstream.internal';"
2026-09-28 23:56:38,020 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:56:38,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:56:38,290 [I] DB_PATCHES have already been applied
I20260928 23:56:38.296679 20832 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHREvfx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:56:38.296751 20832 mini_ranger.cc:193] Using host: 127.20.88.20
I20260928 23:56:38.300602 20832 mini_ranger.cc:227] Ranger admin URL: http://127.20.88.20:45521
Sep 28, 2026 11:56:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:56:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:56:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:56:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:56:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:56:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:56:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:56:39.447031 20832 mini_ranger.cc:160] Time spent starting Ranger: real 9.155s user 0.003s sys 0.007s
Sep 28, 2026 11:56:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:56:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 11:56:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:56:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:57:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:57:01.170593 20832 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.88.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.88.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangerkms_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.88.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/rangerkms_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.88.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.88.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.88.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/HTTP_127.20.88.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:57:01 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639821, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:57:01.263170 20832 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:57:01.274132 20832 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:57:02.367429 20832 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:57:02.367504 20832 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms
I20260928 23:57:02.367518 20832 mini_ranger_kms.cc:227] Using postgres at 127.20.88.20:45189
2026-09-28 23:57:02,555 [I] DB FLAVOR :POSTGRES
2026-09-28 23:57:02,556 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:57:02,556 [I] Checking connection
2026-09-28 23:57:02,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:57:02,806 [I] connection success
2026-09-28 23:57:02,806 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:57:02,806 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:57:02,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:57:03,078 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:57:03,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:57:03,078 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:57:03,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.88.20:45189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:57:03,372 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:57:03.932319 20832 mini_ranger_kms.cc:310] Created kms service
I20260928 23:57:04.161048 20832 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:57:04.208891 20832 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:57:04 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639821, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:04.618964 20832 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:57:04.619131 20832 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHREvfx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHREvfx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:57:04.619155 20832 mini_ranger_kms.cc:240] Using host: 127.20.88.20
I20260928 23:57:04.623389 20832 mini_ranger_kms.cc:276] Ranger KMS PID: 532
I20260928 23:57:04.623488 20832 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.88.20:43641
Sep 28, 2026 11:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:57:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:57:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:57:06.908524 20832 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.645s user 0.012s sys 0.159s
I20260928 23:57:06.908660 20832 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:57:06.908718 20832 mini_ranger_kms.cc:401] 127.20.88.20:43641/kms/v1/keys
Sep 28 23:57:07 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:57:07 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
Sep 28 23:57:07 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639827, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.88.20@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
Sep 28 23:57:10 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:57:10.469036 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42653
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42653
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:10.591131 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:10.591547 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:10.591615 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:10.596000 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:10.596082 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:10.596103 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:10.596120 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:10.596136 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:10.602293 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.88.20:43641/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42653
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:10.603498 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:10.604671 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:10.614244 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:10.614252 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:10.614315 583 server_base.cc:1034] running on GCE node
W20260928 23:57:10.614285 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:10.615146 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:10.616602 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:10.617933 583 hybrid_clock.cc:648] HybridClock initialized: now 1790639830617901 us; error 49 us; skew 500 ppm
Sep 28 23:57:10 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:10.626513 583 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:10.627611 583 webserver.cc:460] Webserver started at http://127.20.88.62:36803/ using document root <none> and password file <none>
I20260928 23:57:10.628223 583 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:10.628364 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:10.628572 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:57:10 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639830, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:10.684767 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "343a3eaadbc24312a1a7ec045627f25e"
format_stamp: "Formatted at 2026-09-28 23:57:10 on dist-test-slave-wvpw"
server_key: "3bd260185d908f8518f83bb8253bbf0f"
server_key_iv: "7f5b9d489006f699c1f2ec96b7d40831"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:10.685446 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "343a3eaadbc24312a1a7ec045627f25e"
format_stamp: "Formatted at 2026-09-28 23:57:10 on dist-test-slave-wvpw"
server_key: "3bd260185d908f8518f83bb8253bbf0f"
server_key_iv: "7f5b9d489006f699c1f2ec96b7d40831"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:10.689816 583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:57:10.705186 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:10.706722 583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:57:10.706887 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "343a3eaadbc24312a1a7ec045627f25e"
format_stamp: "Formatted at 2026-09-28 23:57:10 on dist-test-slave-wvpw"
server_key: "3bd260185d908f8518f83bb8253bbf0f"
server_key_iv: "7f5b9d489006f699c1f2ec96b7d40831"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:10.707036 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:10.719542 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:10.722697 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:10.722932 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:10.731544 583 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42653
I20260928 23:57:10.731560 650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42653 every 8 connection(s)
I20260928 23:57:10.732673 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:10.735637 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:10.738471 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 583
I20260928 23:57:10.738618 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:57:10.738915 20832 external_mini_cluster.cc:1428] Setting key 11f84a3277baa5af32d211920f119525
I20260928 23:57:10.742882 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Bootstrap starting.
I20260928 23:57:10.745787 651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:10.746631 651 log.cc:824] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:10.749255 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: No bootstrap required, opened a new log
Sep 28 23:57:10 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:10.753952 651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:10.754334 651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:10.754426 651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a3eaadbc24312a1a7ec045627f25e, State: Initialized, Role: FOLLOWER
I20260928 23:57:10.754985 651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:10.755159 651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:10.755203 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:10.740526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.20.88.62:42653)
0928 23:57:10.741011 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:10.741032 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:10.741076 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:57:10.741985 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:10.742011 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:10.742455 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:10.742838 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:10.743739 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:10.745273 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:10.746605 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:10.747298 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:10.747484 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:10.750291 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:10.750297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:10.750314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:10.750347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:10.752050 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:10.752580 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:10.752586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:10.752591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:10.752719 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:10.753017 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:10.753024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:10.753026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:10.753475 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:10.753689 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:10.753964 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 23:57:10.754283 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260928 23:57:10.755250 651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:10.755512 651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:10.756816 651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:10.757223 651 leader_election.cc:302] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 343a3eaadbc24312a1a7ec045627f25e; no voters:
I20260928 23:57:10.757615 651 leader_election.cc:288] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:57:10.757897 656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:10.758164 656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 LEADER]: Becoming Leader. State: Replica: 343a3eaadbc24312a1a7ec045627f25e, State: Running, Role: LEADER
I20260928 23:57:10.758546 656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:10.759124 651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:10.760193 658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343a3eaadbc24312a1a7ec045627f25e. Latest consensus state: current_term: 1 leader_uuid: "343a3eaadbc24312a1a7ec045627f25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } } }
I20260928 23:57:10.760345 658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:10.760646 657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "343a3eaadbc24312a1a7ec045627f25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } } }
I20260928 23:57:10.760720 657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:10.761125 665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:10.761441 651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:57:10.767987 665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:10.778699 665 catalog_manager.cc:1347] Generated new cluster ID: 361f6ba3b5044a70870f43be5745eea4
I20260928 23:57:10.778789 665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:10.803524 665 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:57:10.805047 665 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:10.811841 665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Generated new TSK 0
I20260928 23:57:10.812896 665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:57:11.548400 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:11.679251 696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:11.679595 696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:11.679709 696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:11.684324 696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:11.684410 696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:11.684506 696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:11.690872 696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:11.692130 696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:11.693246 696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:11.703617 701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:11.703722 702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:11.703867 704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:11.704068 696 server_base.cc:1034] running on GCE node
I20260928 23:57:11.704591 696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:11.705471 696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:11.706696 696 hybrid_clock.cc:648] HybridClock initialized: now 1790639831706672 us; error 41 us; skew 500 ppm
Sep 28 23:57:11 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:11.710397 696 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:11.711525 696 webserver.cc:460] Webserver started at http://127.20.88.1:41791/ using document root <none> and password file <none>
I20260928 23:57:11.712325 696 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:11.712423 696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:11.712647 696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:57:11 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:11.731050 696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "5c49bc2b00ba4f04bd9443041441a770"
format_stamp: "Formatted at 2026-09-28 23:57:11 on dist-test-slave-wvpw"
server_key: "0d4c32c6de65056ca08f04bcea2057a7"
server_key_iv: "bf73abdce135c1f3384ce9879af08658"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:11.732007 696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "5c49bc2b00ba4f04bd9443041441a770"
format_stamp: "Formatted at 2026-09-28 23:57:11 on dist-test-slave-wvpw"
server_key: "0d4c32c6de65056ca08f04bcea2057a7"
server_key_iv: "bf73abdce135c1f3384ce9879af08658"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:11.750290 696 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.001s sys 0.003s
I20260928 23:57:11.774724 713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:11.776727 696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:57:11.776906 696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "5c49bc2b00ba4f04bd9443041441a770"
format_stamp: "Formatted at 2026-09-28 23:57:11 on dist-test-slave-wvpw"
server_key: "0d4c32c6de65056ca08f04bcea2057a7"
server_key_iv: "bf73abdce135c1f3384ce9879af08658"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:11.777068 696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:11.806124 696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:11.810454 696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:11.810740 696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:11.811830 696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:11.813072 696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:11.813184 696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:11.813311 696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:11.813419 696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:11.827852 696 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:43833
I20260928 23:57:11.827940 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:43833 every 8 connection(s)
I20260928 23:57:11.829038 696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:11.838490 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 696
I20260928 23:57:11.838660 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:57:11.839043 20832 external_mini_cluster.cc:1428] Setting key 276618ecf44f2f468aa52e96c00a7d8d
Sep 28 23:57:11 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:11.848748 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:11.831764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.1:34157)
0928 23:57:11.832495 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:11.832538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:11.833454 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:11.834078 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:11.834086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:11.834361 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:11.834928 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:11.834943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:11.835866 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:11.835869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:11.837281 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:11.837295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:11.838152 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:11.838157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:11.838376 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:11.839164 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:11.839188 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:11.842432 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:11.844941 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:11.844948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:11.844970 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:11.845384 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:11.845744 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:11.845747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:11.845750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:11.845969 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:11.846560 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:11.846568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:11.846967 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 23:57:11.847193 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 23:57:11.847583 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":168,"threads_started":1}
I20260928 23:57:11.849613 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:11.831174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:34157 (local address 127.20.88.62:42653)
0928 23:57:11.831417 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:11.831421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:11.832642 (+ 1221us) server_negotiation.cc:407] Connection header received
0928 23:57:11.833720 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:11.833723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:11.833795 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:11.833918 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:11.835084 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:11.835725 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:11.837461 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:11.837835 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:11.837947 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:11.842668 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:11.842672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:11.842675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:11.842706 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:11.844780 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:11.845529 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:11.845532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:11.845534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:11.845639 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:11.846145 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:11.846149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:11.846151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:11.846406 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:11.846501 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:11.849266 (+ 2765us) server_negotiation.cc:299] Negotiation successful
0928 23:57:11.849433 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260928 23:57:11.855887 828 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:11.856257 828 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:11.857049 828 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:11.859301 615 ts_manager.cc:194] Registered new tserver with Master: 5c49bc2b00ba4f04bd9443041441a770 (127.20.88.1:43833)
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:57:11.861274 615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:34157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:57:11.885777 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:57:11 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:57:12.026461 835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:12.026978 835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:12.027097 835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:12.033524 835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:12.033629 835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:12.033746 835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:12.039680 835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:12.041239 835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:12.042690 835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:12.055708 841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:12.055981 844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:12.056309 835 server_base.cc:1034] running on GCE node
W20260928 23:57:12.055804 842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:12.056995 835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:12.057889 835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:12.059065 835 hybrid_clock.cc:648] HybridClock initialized: now 1790639832059041 us; error 38 us; skew 500 ppm
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:12.062867 835 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:12.063810 835 webserver.cc:460] Webserver started at http://127.20.88.2:39105/ using document root <none> and password file <none>
I20260928 23:57:12.064476 835 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:12.064567 835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:12.064801 835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:12.081173 835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "05c6d6764f614d2eba052c4f9b7123bd"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "57cf776825fed480cc63d23470ab899d"
server_key_iv: "8b234f9bd2325607838a2b8d19fe7310"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.082454 835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "05c6d6764f614d2eba052c4f9b7123bd"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "57cf776825fed480cc63d23470ab899d"
server_key_iv: "8b234f9bd2325607838a2b8d19fe7310"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.093418 835 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.004s
I20260928 23:57:12.114974 851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.117053 835 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260928 23:57:12.117321 835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "05c6d6764f614d2eba052c4f9b7123bd"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "57cf776825fed480cc63d23470ab899d"
server_key_iv: "8b234f9bd2325607838a2b8d19fe7310"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.117583 835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:12.140102 835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:12.144356 835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:12.144635 835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:12.145509 835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:12.146706 835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:12.146834 835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.146947 835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:12.147018 835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.157615 835 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:33985
I20260928 23:57:12.158984 835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:12.164433 964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:33985 every 8 connection(s)
I20260928 23:57:12.167215 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 835
I20260928 23:57:12.167376 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:57:12.167738 20832 external_mini_cluster.cc:1428] Setting key 7de55d420fd4feaae649f81e5a81a3b7
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:57:12.185431 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:12.168928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:51463 (local address 127.20.88.62:42653)
0928 23:57:12.169112 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:12.169117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:12.169634 (+ 517us) server_negotiation.cc:407] Connection header received
0928 23:57:12.171193 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:12.171200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:12.171273 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:12.171334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:12.173345 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.174154 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.175986 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.176390 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.176522 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:12.180156 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:12.180161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:12.180164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:12.180205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:12.182295 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:12.183158 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:12.183162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:12.183165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:12.183275 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:12.183780 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:12.183784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:12.183786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:12.184053 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:12.184135 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:12.185051 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 23:57:12.185224 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 23:57:12.187656 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:12.167611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.2:51463)
0928 23:57:12.169461 (+ 1850us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:12.169534 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:12.170901 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:12.171856 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:12.171868 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:12.172308 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:12.173146 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:12.173170 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.174339 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.174343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:12.175799 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:12.175813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.176751 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.176756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:12.176988 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:12.177796 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:12.177821 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:12.179864 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:12.182509 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:12.182517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:12.182540 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:12.183007 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:12.183414 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:12.183417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:12.183419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:12.183634 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:12.184224 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:12.184269 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:12.184698 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 23:57:12.186054 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0928 23:57:12.186473 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":180,"threads_started":1}
I20260928 23:57:12.189801 965 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:12.190276 965 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:12.190959 965 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:57:12.192314 615 ts_manager.cc:194] Registered new tserver with Master: 05c6d6764f614d2eba052c4f9b7123bd (127.20.88.2:33985)
I20260928 23:57:12.193109 615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:51463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:57:12.217608 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:12.405992 970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:12.406488 970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:12.406610 970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:12.413447 970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:12.413612 970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:12.413754 970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:12.422106 970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:12.423807 970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:12.425350 970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:12.439006 978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:12.439006 976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:12.439800 975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:12.441305 970 server_base.cc:1034] running on GCE node
I20260928 23:57:12.442006 970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:12.442940 970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:12.458717 970 hybrid_clock.cc:648] HybridClock initialized: now 1790639832458631 us; error 77 us; skew 500 ppm
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:12.462975 970 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:12.464392 970 webserver.cc:460] Webserver started at http://127.20.88.3:42747/ using document root <none> and password file <none>
I20260928 23:57:12.465310 970 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:12.465476 970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:12.465812 970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639831, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:12.488140 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "d1ff41d72c074fd69365f93fdde40429"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "15e8c57d57e50e544886587afbebae4a"
server_key_iv: "33ffd611e23f36331c15984e94865304"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.496429 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "d1ff41d72c074fd69365f93fdde40429"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "15e8c57d57e50e544886587afbebae4a"
server_key_iv: "33ffd611e23f36331c15984e94865304"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.516548 970 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.005s sys 0.001s
I20260928 23:57:12.555428 985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.557616 970 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.002s sys 0.000s
I20260928 23:57:12.557801 970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "d1ff41d72c074fd69365f93fdde40429"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "15e8c57d57e50e544886587afbebae4a"
server_key_iv: "33ffd611e23f36331c15984e94865304"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.557960 970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:12.588644 970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:12.593099 970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:12.593436 970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:12.594261 970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:12.595630 970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:12.595755 970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 23:57:12.595932 970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:12.596006 970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.606763 970 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:37899
I20260928 23:57:12.608101 970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:12.613852 1098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:37899 every 8 connection(s)
I20260928 23:57:12.615408 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 970
I20260928 23:57:12.615553 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:57:12.615891 20832 external_mini_cluster.cc:1428] Setting key 3fc2ef577dcf247e62ac7250d1c18460
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:12.637805 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:12.620853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:48639 (local address 127.20.88.62:42653)
0928 23:57:12.620985 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:12.620989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:12.623078 (+ 2089us) server_negotiation.cc:407] Connection header received
0928 23:57:12.624440 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:12.624445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:12.624506 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:12.624563 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:12.626322 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.627162 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.628899 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.629219 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.629295 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:12.632759 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:12.632764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:12.632767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:12.632801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:12.634633 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:12.635387 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:12.635391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:12.635410 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:12.635513 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:12.636032 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:12.636035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:12.636038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:12.636361 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:12.636468 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:12.637422 (+ 954us) server_negotiation.cc:299] Negotiation successful
0928 23:57:12.637591 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:57:12.640266 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:12.621483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.3:48639)
0928 23:57:12.622863 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:12.622965 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:12.624146 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:12.625026 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:12.625036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:12.625429 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:12.626144 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:12.626164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.627333 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.627336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:12.628738 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:12.628750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:12.629522 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:12.629526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:12.629742 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:12.630572 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:12.630594 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:12.632528 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:12.634806 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:12.634813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:12.634835 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:12.635248 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:12.635640 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:12.635644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:12.635646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:12.635901 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:12.636555 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:12.636565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:12.637061 (+ 496us) client_negotiation.cc:769] Sending connection context
0928 23:57:12.638408 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0928 23:57:12.638905 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":195,"threads_started":1}
I20260928 23:57:12.642297 1099 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:12.642566 1099 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:12.643169 1099 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:12.644414 614 ts_manager.cc:194] Registered new tserver with Master: d1ff41d72c074fd69365f93fdde40429 (127.20.88.3:37899)
I20260928 23:57:12.645184 614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:48639
I20260928 23:57:12.655941 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:57:12.656987 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 696
I20260928 23:57:12.665107 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 835
I20260928 23:57:12.675559 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 970
I20260928 23:57:12.682276 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 583
I20260928 23:57:12.690505 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42653
--webserver_interface=127.20.88.62
--webserver_port=36803
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:42653
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:12.808251 1102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:12.808619 1102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:12.808693 1102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:12.813124 1102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:12.813212 1102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:12.813232 1102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:12.813249 1102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:12.813265 1102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:12.819347 1102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:42653
--ranger_config_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:42653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=36803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:12.820621 1102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:12.821736 1102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:12.830860 1108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:12.830861 1107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:12.830875 1110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:12.831005 1102 server_base.cc:1034] running on GCE node
I20260928 23:57:12.831815 1102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:12.833078 1102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:12.834249 1102 hybrid_clock.cc:648] HybridClock initialized: now 1790639832834231 us; error 35 us; skew 500 ppm
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:12.837330 1102 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:12.838289 1102 webserver.cc:460] Webserver started at http://127.20.88.62:36803/ using document root <none> and password file <none>
I20260928 23:57:12.838866 1102 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:12.838940 1102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:57:12 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639832, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:12.890530 1102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:12.893021 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:12.894436 1102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:12.894586 1102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "343a3eaadbc24312a1a7ec045627f25e"
format_stamp: "Formatted at 2026-09-28 23:57:10 on dist-test-slave-wvpw"
server_key: "3bd260185d908f8518f83bb8253bbf0f"
server_key_iv: "7f5b9d489006f699c1f2ec96b7d40831"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:12.895172 1102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:12.910621 1102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:12.913879 1102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:12.914103 1102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:12.922019 1102 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:42653
I20260928 23:57:12.922025 1170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:42653 every 8 connection(s)
I20260928 23:57:12.923209 1102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:12.927894 1171 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:57:12.928907 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1102
I20260928 23:57:12.929368 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:43833
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:42653
--webserver_port=41791
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:12.929982 1171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Bootstrap starting.
I20260928 23:57:12.947412 1171 log.cc:824] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:12.954239 1171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:57:12.954670 1171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Bootstrap complete.
I20260928 23:57:12.958927 1171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:12.959340 1171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343a3eaadbc24312a1a7ec045627f25e, State: Initialized, Role: FOLLOWER
I20260928 23:57:12.959859 1171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:12.959992 1171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:12.960081 1171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:12.960192 1171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:57:12.962262 1171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:12.962652 1171 leader_election.cc:302] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 343a3eaadbc24312a1a7ec045627f25e; no voters:
I20260928 23:57:12.963042 1171 leader_election.cc:288] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:57:12.963132 1175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:57:12.963353 1175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [term 2 LEADER]: Becoming Leader. State: Replica: 343a3eaadbc24312a1a7ec045627f25e, State: Running, Role: LEADER
I20260928 23:57:12.963666 1175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } }
I20260928 23:57:12.964365 1171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:12.964684 1175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 343a3eaadbc24312a1a7ec045627f25e. Latest consensus state: current_term: 2 leader_uuid: "343a3eaadbc24312a1a7ec045627f25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } } }
I20260928 23:57:12.964803 1175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:12.965214 1176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "343a3eaadbc24312a1a7ec045627f25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "343a3eaadbc24312a1a7ec045627f25e" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 42653 } } }
I20260928 23:57:12.965392 1176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:12.967736 1179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:12.968207 1171 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:57:12.971017 1179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:12.971680 1179 catalog_manager.cc:1259] Loaded cluster ID: 361f6ba3b5044a70870f43be5745eea4
I20260928 23:57:12.971746 1179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:12.977735 1179 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:12.979110 1179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Loaded TSK: 0
I20260928 23:57:12.980590 1179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:57:13.058308 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:13.058650 1173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:13.058730 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:13.063628 1173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:13.063714 1173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:13.063832 1173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:13.070472 1173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:43833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=41791
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:13.071720 1173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:13.072902 1173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:13.085762 1211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:13.085913 1208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:13.086289 1209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:13.086648 1173 server_base.cc:1034] running on GCE node
I20260928 23:57:13.087127 1173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:13.087910 1173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:13.089093 1173 hybrid_clock.cc:648] HybridClock initialized: now 1790639833089019 us; error 90 us; skew 500 ppm
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:13.092553 1173 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:13.093647 1173 webserver.cc:460] Webserver started at http://127.20.88.1:41791/ using document root <none> and password file <none>
I20260928 23:57:13.094264 1173 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:13.094372 1173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:13.131747 1173 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 23:57:13.134925 1219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.136555 1173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:57:13.136735 1173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "5c49bc2b00ba4f04bd9443041441a770"
format_stamp: "Formatted at 2026-09-28 23:57:11 on dist-test-slave-wvpw"
server_key: "0d4c32c6de65056ca08f04bcea2057a7"
server_key_iv: "bf73abdce135c1f3384ce9879af08658"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:13.137467 1173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:13.154291 1173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:13.158097 1173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:13.158356 1173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:13.159077 1173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:13.160048 1173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:13.160115 1173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.160183 1173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:13.160211 1173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.172158 1173 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:43833
I20260928 23:57:13.173715 1173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:13.178911 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1173
I20260928 23:57:13.179400 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:33985
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:42653
--webserver_port=39105
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:13.181631 1332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:43833 every 8 connection(s)
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:13.223016 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.184365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.1:60115)
0928 23:57:13.191366 (+ 7001us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:13.191458 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:13.192705 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:13.198208 (+ 5503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:13.198221 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:13.198716 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:13.199378 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.199397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.201538 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.201543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.203028 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.203044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.213690 (+ 10646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.213696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.213912 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.214672 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:13.214694 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:13.216453 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:13.219152 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.219159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.219174 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.219562 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.219979 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.219982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.219985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.220168 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.221034 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:13.221040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:13.221382 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 23:57:13.221576 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 23:57:13.221931 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6657,"thread_start_us":158,"threads_started":1}
I20260928 23:57:13.224888 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.184120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:60115 (local address 127.20.88.62:42653)
0928 23:57:13.193348 (+ 9228us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:13.193372 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:13.193422 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:57:13.193779 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:13.193796 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:13.194298 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:13.194748 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:13.199661 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.201329 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.212373 (+ 11044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.213195 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.213417 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.216716 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:13.216723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:13.216740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:13.216774 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:13.218986 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.219711 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.219718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.219724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.219867 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.220336 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.220340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.220342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.220843 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:13.220976 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:13.223600 (+ 2624us) server_negotiation.cc:299] Negotiation successful
0928 23:57:13.223915 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9014,"thread_start_us":152,"threads_started":1}
I20260928 23:57:13.227634 1333 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:13.227928 1333 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.228623 1333 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.231307 1333 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:57:13.231465 1333 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.231705 1333 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:13.232760 1333 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.232980 1333 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.233872 1333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:57:13.344655 1335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:13.345094 1335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:13.345165 1335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:13.350574 1335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:13.350668 1335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:13.350759 1335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:13.357995 1335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:33985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=39105
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:13.359473 1335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:13.360977 1335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:13.373072 1343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:13.373063 1345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:13.373701 1335 server_base.cc:1034] running on GCE node
W20260928 23:57:13.373987 1342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:13.374444 1335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:13.375178 1335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:13.376343 1335 hybrid_clock.cc:648] HybridClock initialized: now 1790639833376314 us; error 48 us; skew 500 ppm
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:13.379606 1335 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:13.380972 1335 webserver.cc:460] Webserver started at http://127.20.88.2:39105/ using document root <none> and password file <none>
I20260928 23:57:13.381806 1335 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:13.381932 1335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:13.406337 1335 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 23:57:13.410259 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.412000 1335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:57:13.412420 1335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "05c6d6764f614d2eba052c4f9b7123bd"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "57cf776825fed480cc63d23470ab899d"
server_key_iv: "8b234f9bd2325607838a2b8d19fe7310"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:13.413225 1335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:13.470661 1335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:13.474637 1335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:13.474900 1335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:13.475739 1335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:13.477003 1335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:13.477067 1335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.477126 1335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:13.477149 1335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.490453 1335 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:33985
I20260928 23:57:13.490634 1466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:33985 every 8 connection(s)
I20260928 23:57:13.492154 1335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:13.494185 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1335
I20260928 23:57:13.494747 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.88.20:43641/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:37899
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:42653
--webserver_port=42747
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:13.511307 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.496384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:37857 (local address 127.20.88.62:42653)
0928 23:57:13.497748 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:13.497753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:13.497773 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:57:13.498809 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:13.498814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:13.498877 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:13.498972 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:13.500448 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.501041 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.502416 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.502658 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.502780 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.506192 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:13.506197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:13.506200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:13.506238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:13.508291 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.509384 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.509388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.509390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.509491 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.509986 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.509989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.509991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.510202 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:13.510327 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:13.510953 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 23:57:13.511108 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246}
I20260928 23:57:13.512488 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.496651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.2:37857)
0928 23:57:13.497199 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:13.497259 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:13.498458 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:13.499233 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:13.499258 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:13.499572 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:13.500177 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.500193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.501218 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.501222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.502213 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.502228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.502831 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.502837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.503072 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.503858 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:13.503882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:13.505924 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:13.508779 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.508787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.508810 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.509231 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.509612 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.509616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.509618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.509844 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.510343 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:13.510353 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:13.510673 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 23:57:13.510895 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 23:57:13.511258 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":161,"threads_started":1}
I20260928 23:57:13.513980 1467 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:13.514258 1467 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.514938 1467 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.516871 1467 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:57:13.517022 1467 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.517251 1467 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:13.518285 1467 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.518810 1467 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.520061 1467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:57:13.643146 1469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:13.643476 1469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:13.643601 1469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:13.648461 1469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:13.648553 1469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:13.648660 1469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:13.656455 1469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.88.20:43641/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:37899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=42747
--tserver_master_addrs=127.20.88.62:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:13.658162 1469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:13.659595 1469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:13.674002 1479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:13.674418 1477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:13.675148 1476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:13.675917 1469 server_base.cc:1034] running on GCE node
I20260928 23:57:13.676527 1469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:13.677479 1469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:13.678696 1469 hybrid_clock.cc:648] HybridClock initialized: now 1790639833678670 us; error 39 us; skew 500 ppm
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:13.682812 1469 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:13.684269 1469 webserver.cc:460] Webserver started at http://127.20.88.3:42747/ using document root <none> and password file <none>
I20260928 23:57:13.685091 1469 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:13.685236 1469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:13.716038 1469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:13.719156 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.720824 1469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:57:13.720983 1469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "d1ff41d72c074fd69365f93fdde40429"
format_stamp: "Formatted at 2026-09-28 23:57:12 on dist-test-slave-wvpw"
server_key: "15e8c57d57e50e544886587afbebae4a"
server_key_iv: "33ffd611e23f36331c15984e94865304"
server_key_version: "kuduclusterkey@0"
I20260928 23:57:13.721613 1469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:13.748632 1469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:13.752920 1469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:13.753157 1469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:13.753978 1469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:13.755193 1469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:13.755264 1469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.755347 1469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:13.755396 1469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:13.771329 1469 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:37899
I20260928 23:57:13.772792 1469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:13.773451 1600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:37899 every 8 connection(s)
I20260928 23:57:13.777513 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1469
Sep 28 23:57:13 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639833, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:13.789400 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.778567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.20.88.62:42653)
0928 23:57:13.778840 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:13.778845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:13.778858 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:57:13.778902 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:13.778906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:13.778957 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:13.779060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:13.780914 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.781703 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.782720 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.783072 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.783170 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.784694 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:13.784698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:13.784701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:13.784733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:13.786574 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.787083 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.787091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.787094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.787177 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.788030 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.788032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.788034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.788333 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:13.788485 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:13.789012 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 23:57:13.789150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260928 23:57:13.789601 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.775761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:55391 (local address 127.20.88.62:42653)
0928 23:57:13.775930 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:13.775936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:13.776968 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 23:57:13.778134 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:13.778138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:13.778213 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:13.778326 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:13.779911 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.780758 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.782075 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.782428 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.782538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.785535 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:13.785540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:13.785543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:13.785597 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:13.787149 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.787789 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.787791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.787792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.787856 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:13.788301 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:13.788305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:13.788307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:13.788547 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:13.788656 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:13.789276 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 23:57:13.789381 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":3328}
I20260928 23:57:13.790977 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:13.776225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:42653 (local address 127.20.88.3:55391)
0928 23:57:13.776785 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:13.776840 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:13.777849 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:13.778560 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:13.778571 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:13.778939 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:13.779733 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.779756 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.780893 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.780896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.781948 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:13.781960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:13.782598 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:13.782602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:13.782819 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:13.783519 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:13.783543 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:13.785340 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:13.787273 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.787286 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.787306 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.787696 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.787948 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:13.787952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:13.787954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:13.788151 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:13.788662 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:13.788667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:13.788991 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 23:57:13.789453 (+ 462us) client_negotiation.cc:240] Negotiation successful
0928 23:57:13.789848 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":152,"threads_started":1}
I20260928 23:57:13.792197 1601 heartbeater.cc:347] Connected to a master server at 127.20.88.62:42653
I20260928 23:57:13.792491 1601 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.793040 1601 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.794705 1601 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:42653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:57:13.794903 1601 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.795106 1601 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:13.796053 1601 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:13.796315 1601 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
W20260928 23:57:13.798990 1601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:57:14 dist-test-slave-wvpw krb5kdc[32607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639834, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:14.234148 1333 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:14.234510 1333 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:14.235973 1135 ts_manager.cc:194] Registered new tserver with Master: 5c49bc2b00ba4f04bd9443041441a770 (127.20.88.1:43833)
I20260928 23:57:14.237679 1135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:60115
I20260928 23:57:14.520360 1467 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:14.520714 1467 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:14.521759 1135 ts_manager.cc:194] Registered new tserver with Master: 05c6d6764f614d2eba052c4f9b7123bd (127.20.88.2:33985)
I20260928 23:57:14.522433 1135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:37857
Sep 28 23:57:14 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639834, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:14.799302 1601 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:14.799674 1601 heartbeater.cc:511] Master 127.20.88.62:42653 requested a full tablet report, sending...
I20260928 23:57:14.800627 1135 ts_manager.cc:194] Registered new tserver with Master: d1ff41d72c074fd69365f93fdde40429 (127.20.88.3:37899)
I20260928 23:57:14.801528 1135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:55391
I20260928 23:57:14.803092 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:57:14.815244 1622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.804874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.20.88.62:42653)
0928 23:57:14.805191 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:14.805197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:14.805541 (+ 344us) server_negotiation.cc:407] Connection header received
0928 23:57:14.805656 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:14.805659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:14.805713 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:14.805803 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:14.806762 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.807642 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.808956 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.809345 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.809463 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.811399 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:14.811404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:14.811406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:14.811438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:14.813279 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:14.813815 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:14.813820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:14.813822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:14.813930 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:14.814238 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:14.814241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:14.814243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:14.814463 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:14.814563 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:14.814926 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 23:57:14.815070 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
I20260928 23:57:14.820659 1135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:57:14.825765 1135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:57:14.850241 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.841974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:37899 (local address 127.0.0.1:36394)
0928 23:57:14.844395 (+ 2421us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:14.844411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:14.844516 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:14.845352 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:14.845359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:14.845385 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:14.845532 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.845540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.846942 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.846946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.848387 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.848398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.849678 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.849681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.849793 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.849830 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:57:14.849947 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:57:14.850078 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2350,"thread_start_us":83,"threads_started":1}
Sep 28 23:57:14 dist-test-slave-wvpw krb5kdc[32607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639834, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for HTTP/127.20.88.20@KRBTEST.COM
I20260928 23:57:14.850687 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.842151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.20.88.3:37899)
0928 23:57:14.842434 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:14.842442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:14.844625 (+ 2183us) server_negotiation.cc:407] Connection header received
0928 23:57:14.844721 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:14.844727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:14.844993 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:14.845141 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:14.845667 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.846790 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.848532 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.849358 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.849465 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.849616 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:14.850444 (+ 828us) server_negotiation.cc:299] Negotiation successful
0928 23:57:14.850556 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260928 23:57:14.854952 1518 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c0084548f463290301ffbb9a78d8e (DEFAULT_TABLE table=test-table [id=f4a0e40be20f4363aa5fdcbf690bb72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:14.857049 1518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c0084548f463290301ffbb9a78d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:14.857350 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.840739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:43833 (local address 127.0.0.1:38238)
0928 23:57:14.841210 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:14.841436 (+ 226us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:14.841586 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:14.852932 (+ 11346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:14.852935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:14.852954 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:14.853051 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.853057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.854407 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.854411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.855537 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.855552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.856916 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.856920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.857014 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.857044 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:57:14.857102 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:57:14.857186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":142,"threads_started":1}
I20260928 23:57:14.858346 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.845283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.20.88.1:43833)
0928 23:57:14.852486 (+ 7203us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:14.852494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:14.852520 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:57:14.852605 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:14.852610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:14.852789 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:14.852966 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:14.853176 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.854267 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.855714 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.856740 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.857875 (+ 1135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.858043 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:14.858100 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:57:14.858201 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7078,"thread_start_us":96,"threads_started":1}
I20260928 23:57:14.861708 1267 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c0084548f463290301ffbb9a78d8e (DEFAULT_TABLE table=test-table [id=f4a0e40be20f4363aa5fdcbf690bb72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:14.862900 1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c0084548f463290301ffbb9a78d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:14.874949 1635 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Bootstrap starting.
I20260928 23:57:14.877771 1635 tablet_bootstrap.cc:654] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:14.879015 1635 log.cc:824] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:14.881947 1635 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: No bootstrap required, opened a new log
I20260928 23:57:14.882177 1635 ts_tablet_manager.cc:1397] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:57:14.886307 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.841623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:33985 (local address 127.0.0.1:55722)
0928 23:57:14.842036 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:14.842052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:14.842143 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:14.847328 (+ 5185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:14.847332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:14.847352 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:14.847459 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.847468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.850958 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.850961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.852256 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:14.852268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.885803 (+ 33535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.885808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:14.885929 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.885960 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:57:14.886044 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:57:14.886142 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
I20260928 23:57:14.886584 1635 raft_consensus.cc:348] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.887157 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:14.841799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.20.88.2:33985)
0928 23:57:14.846806 (+ 5007us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:14.846814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:14.846839 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:57:14.846921 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:14.846927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:14.847155 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:14.847632 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:14.847810 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.849924 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.884690 (+ 34766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:14.885643 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:14.885780 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:14.885941 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:14.886886 (+ 945us) server_negotiation.cc:299] Negotiation successful
0928 23:57:14.887006 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4884,"thread_start_us":97,"threads_started":1}
I20260928 23:57:14.887461 1635 raft_consensus.cc:374] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:14.887581 1635 raft_consensus.cc:729] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c49bc2b00ba4f04bd9443041441a770, State: Initialized, Role: FOLLOWER
I20260928 23:57:14.888211 1635 consensus_queue.cc:260] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.889760 1635 ts_tablet_manager.cc:1428] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260928 23:57:14.890201 1333 heartbeater.cc:503] Master 127.20.88.62:42653 was elected leader, sending a full tablet report...
I20260928 23:57:14.890576 1401 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7c0084548f463290301ffbb9a78d8e (DEFAULT_TABLE table=test-table [id=f4a0e40be20f4363aa5fdcbf690bb72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:14.891808 1401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7c0084548f463290301ffbb9a78d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:14.895843 1637 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Bootstrap starting.
I20260928 23:57:14.898600 1637 tablet_bootstrap.cc:654] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:14.899658 1637 log.cc:824] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:14.902745 1637 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: No bootstrap required, opened a new log
I20260928 23:57:14.903463 1637 ts_tablet_manager.cc:1397] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 23:57:14.904663 1640 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Bootstrap starting.
I20260928 23:57:14.909247 1637 raft_consensus.cc:348] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.909525 1640 tablet_bootstrap.cc:654] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:14.910100 1637 raft_consensus.cc:374] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:14.910746 1640 log.cc:824] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:14.912987 1637 raft_consensus.cc:729] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ff41d72c074fd69365f93fdde40429, State: Initialized, Role: FOLLOWER
I20260928 23:57:14.914386 1637 consensus_queue.cc:260] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.915390 1601 heartbeater.cc:503] Master 127.20.88.62:42653 was elected leader, sending a full tablet report...
I20260928 23:57:14.919196 1640 tablet_bootstrap.cc:492] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: No bootstrap required, opened a new log
I20260928 23:57:14.919441 1640 ts_tablet_manager.cc:1397] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260928 23:57:14.919845 1637 ts_tablet_manager.cc:1428] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Time spent starting tablet: real 0.016s user 0.002s sys 0.004s
I20260928 23:57:14.925617 1640 raft_consensus.cc:348] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.926183 1640 raft_consensus.cc:374] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:14.926295 1640 raft_consensus.cc:729] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c6d6764f614d2eba052c4f9b7123bd, State: Initialized, Role: FOLLOWER
I20260928 23:57:14.928644 1640 consensus_queue.cc:260] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:14.931332 1640 ts_tablet_manager.cc:1428] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260928 23:57:14.931844 1467 heartbeater.cc:503] Master 127.20.88.62:42653 was elected leader, sending a full tablet report...
W20260928 23:57:14.937364 1334 tablet.cc:2369] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:57:14.997040 1468 tablet.cc:2369] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:57:15.025210 1602 tablet.cc:2369] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:57:15.036721 1643 raft_consensus.cc:482] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:15.037007 1643 raft_consensus.cc:504] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:15.038486 1643 leader_election.cc:288] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1ff41d72c074fd69365f93fdde40429 (127.20.88.3:37899), 5c49bc2b00ba4f04bd9443041441a770 (127.20.88.1:43833)
I20260928 23:57:15.044795 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.039251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:50321 (local address 127.20.88.3:37899)
0928 23:57:15.039536 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:15.039541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:15.039960 (+ 419us) server_negotiation.cc:407] Connection header received
0928 23:57:15.040163 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:15.040167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:15.040219 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:15.040305 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:15.040742 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.041570 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.043328 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.043962 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.044045 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:15.044117 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:15.044560 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 23:57:15.044648 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260928 23:57:15.045490 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.038736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:37899 (local address 127.20.88.2:50321)
0928 23:57:15.039858 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:15.039873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:15.040042 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:15.040454 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:15.040456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:15.040474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:15.040616 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:15.040622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.041719 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.041723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:15.043139 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:15.043150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.044290 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.044294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:15.044406 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:15.044430 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:57:15.045224 (+ 794us) client_negotiation.cc:240] Negotiation successful
0928 23:57:15.045337 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":117,"threads_started":1}
I20260928 23:57:15.046353 1540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c0084548f463290301ffbb9a78d8e" candidate_uuid: "05c6d6764f614d2eba052c4f9b7123bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ff41d72c074fd69365f93fdde40429" is_pre_election: true
I20260928 23:57:15.046703 1540 raft_consensus.cc:2454] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c6d6764f614d2eba052c4f9b7123bd in term 0.
I20260928 23:57:15.048170 1355 leader_election.cc:302] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c6d6764f614d2eba052c4f9b7123bd, d1ff41d72c074fd69365f93fdde40429; no voters:
I20260928 23:57:15.049188 1643 raft_consensus.cc:2781] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:57:15.049263 1643 raft_consensus.cc:482] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:57:15.049314 1643 raft_consensus.cc:3037] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:15.050659 1643 raft_consensus.cc:504] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:15.051043 1643 leader_election.cc:288] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [CANDIDATE]: Term 1 election: Requested vote from peers d1ff41d72c074fd69365f93fdde40429 (127.20.88.3:37899), 5c49bc2b00ba4f04bd9443041441a770 (127.20.88.1:43833)
I20260928 23:57:15.051712 1540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c0084548f463290301ffbb9a78d8e" candidate_uuid: "05c6d6764f614d2eba052c4f9b7123bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ff41d72c074fd69365f93fdde40429"
I20260928 23:57:15.051848 1540 raft_consensus.cc:3037] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:15.053145 1540 raft_consensus.cc:2454] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c6d6764f614d2eba052c4f9b7123bd in term 1.
I20260928 23:57:15.053570 1355 leader_election.cc:302] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c6d6764f614d2eba052c4f9b7123bd, d1ff41d72c074fd69365f93fdde40429; no voters:
I20260928 23:57:15.053799 1643 raft_consensus.cc:2781] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:15.054054 1643 raft_consensus.cc:686] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [term 1 LEADER]: Becoming Leader. State: Replica: 05c6d6764f614d2eba052c4f9b7123bd, State: Running, Role: LEADER
I20260928 23:57:15.054502 1643 consensus_queue.cc:237] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } }
I20260928 23:57:15.058082 1135 catalog_manager.cc:5495] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 05c6d6764f614d2eba052c4f9b7123bd (127.20.88.2). New cstate: current_term: 1 leader_uuid: "05c6d6764f614d2eba052c4f9b7123bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c6d6764f614d2eba052c4f9b7123bd" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 33985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 } health_report { overall_health: UNKNOWN } } }
I20260928 23:57:15.065671 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.039400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:58503 (local address 127.20.88.1:43833)
0928 23:57:15.039625 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:15.039629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:15.048548 (+ 8919us) server_negotiation.cc:407] Connection header received
0928 23:57:15.048678 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:15.048681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:15.048737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:15.048810 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:15.059818 (+ 11008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.061663 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.064032 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.064804 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.064903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:15.064978 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:15.065410 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 23:57:15.065525 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260928 23:57:15.066478 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.038869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:43833 (local address 127.20.88.2:58503)
0928 23:57:15.048418 (+ 9549us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:15.048434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:15.048542 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:15.059451 (+ 10909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:15.059456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:15.059481 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:15.059639 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:15.059647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.062570 (+ 2923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.062574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:15.063826 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:15.063839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.065120 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.065123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:15.065238 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:15.065261 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:57:15.066125 (+ 864us) client_negotiation.cc:240] Negotiation successful
0928 23:57:15.066221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9463,"thread_start_us":94,"threads_started":1}
I20260928 23:57:15.067471 1286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c0084548f463290301ffbb9a78d8e" candidate_uuid: "05c6d6764f614d2eba052c4f9b7123bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c49bc2b00ba4f04bd9443041441a770" is_pre_election: true
I20260928 23:57:15.067853 1286 raft_consensus.cc:2454] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c6d6764f614d2eba052c4f9b7123bd in term 0.
I20260928 23:57:15.068287 1285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7c0084548f463290301ffbb9a78d8e" candidate_uuid: "05c6d6764f614d2eba052c4f9b7123bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c49bc2b00ba4f04bd9443041441a770"
I20260928 23:57:15.068424 1285 raft_consensus.cc:3037] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:15.069880 1285 raft_consensus.cc:2454] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c6d6764f614d2eba052c4f9b7123bd in term 1.
I20260928 23:57:15.110491 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.104901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.20.88.2:33985)
0928 23:57:15.105018 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:15.105023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:15.105040 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:57:15.105150 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:15.105153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:15.105249 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:15.105386 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:57:15.105843 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.106624 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.107960 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:15.108349 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:15.108451 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:15.108919 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:57:15.109569 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:57:15.109693 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:15.110221 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 23:57:15.110300 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:57:15.124424 1285 raft_consensus.cc:1261] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 1 FOLLOWER]: Refusing update from remote peer 05c6d6764f614d2eba052c4f9b7123bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:15.125476 1647 consensus_queue.cc:1035] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c49bc2b00ba4f04bd9443041441a770" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:15.134087 1540 raft_consensus.cc:1261] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 1 FOLLOWER]: Refusing update from remote peer 05c6d6764f614d2eba052c4f9b7123bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:15.135066 1647 consensus_queue.cc:1035] T 5b7c0084548f463290301ffbb9a78d8e P 05c6d6764f614d2eba052c4f9b7123bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ff41d72c074fd69365f93fdde40429" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:15.149715 1650 mvcc.cc:205] Tried to move back new op lower bound from 7334460764658483200 to 7334460764386504704. Current Snapshot: MvccSnapshot[applied={T|T < 7334460764658483200}]
I20260928 23:57:15.165844 1135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:15.166060 1135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:15.167361 1653 mvcc.cc:205] Tried to move back new op lower bound from 7334460764658483200 to 7334460764386504704. Current Snapshot: MvccSnapshot[applied={T|T < 7334460764658483200}]
I20260928 23:57:15.169219 1135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 343a3eaadbc24312a1a7ec045627f25e: Sending DeleteTablet for 3 replicas of tablet 5b7c0084548f463290301ffbb9a78d8e
I20260928 23:57:15.170377 1267 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c0084548f463290301ffbb9a78d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:15 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:38238
I20260928 23:57:15.170383 1518 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c0084548f463290301ffbb9a78d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:15 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:36394
I20260928 23:57:15.170948 1401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7c0084548f463290301ffbb9a78d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:15 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55722
I20260928 23:57:15.171818 1660 tablet_replica.cc:330] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770: stopping tablet replica
I20260928 23:57:15.172195 1660 raft_consensus.cc:2229] T 5b7c0084548f463290301ffbb9a78d8e P 5c49bc2b00ba4f04bd9443041441a770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:15.172258 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1173
W20260928 23:57:15.182137 1119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:43833 (error 108)
I20260928 23:57:15.183805 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.183506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:43833 (local address 127.0.0.1:38250)
0928 23:57:15.183621 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:15.183693 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:57:15.183784 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1335
W20260928 23:57:15.184074 1119 catalog_manager.cc:4584] TS 5c49bc2b00ba4f04bd9443041441a770 (127.20.88.1:43833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b7c0084548f463290301ffbb9a78d8e: Network error: Client connection negotiation failed: client connection to 127.20.88.1:43833: connect: Connection refused (error 111)
I20260928 23:57:15.186760 1659 tablet_replica.cc:330] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: stopping tablet replica
I20260928 23:57:15.187183 1659 raft_consensus.cc:2229] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:15.187608 1659 raft_consensus.cc:2258] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:15.190130 1659 ts_tablet_manager.cc:1905] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:15.193348 1659 ts_tablet_manager.cc:1918] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:15.193572 1659 log.cc:1196] T 5b7c0084548f463290301ffbb9a78d8e P d1ff41d72c074fd69365f93fdde40429: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/5b7c0084548f463290301ffbb9a78d8e
I20260928 23:57:15.194568 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1469
I20260928 23:57:15.194792 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:15.194460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:33985 (local address 127.0.0.1:55738)
0928 23:57:15.194651 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:15.194704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 23:57:15.203140 1630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:57:15.202483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:37899 (local address 127.0.0.1:36404)
0928 23:57:15.202674 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:15.202690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:15.202791 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:15.202995 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.3:37899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 23:57:15.203982 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1102
I20260928 23:57:15.216552 20832 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:57:15.561913 20832 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:57:15.562036 20832 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:57:15.907893 20832 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:57:15.908 UTC [32637] LOG: received smart shutdown request
2026-09-28 23:57:15.912 UTC [32637] LOG: background worker "logical replication launcher" (PID 32646) exited with exit code 1
2026-09-28 23:57:15.913 UTC [32641] LOG: shutting down
2026-09-28 23:57:16.105 UTC [32637] LOG: database system is shut down
2026-09-28T23:57:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50967 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:57:16 dist-test-slave-wvpw krb5kdc[1671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:57:16 dist-test-slave-wvpw krb5kdc[1671](info): set up 2 sockets
Sep 28 23:57:16 dist-test-slave-wvpw krb5kdc[1671](info): commencing operation
krb5kdc: starting...
W20260928 23:57:17.277040 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:57:17.405107 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37187
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:17.519799 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:17.520174 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:17.520289 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:17.524642 1684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:17.524760 1684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:17.524797 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:17.524837 1684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:17.524878 1684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:17.530496 1684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:17.531760 1684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:17.532894 1684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:17.542066 1689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:17.542078 1692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:17.542068 1690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:17.542541 1684 server_base.cc:1034] running on GCE node
I20260928 23:57:17.543110 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:17.544447 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:17.545643 1684 hybrid_clock.cc:648] HybridClock initialized: now 1790639837545636 us; error 36 us; skew 500 ppm
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:17.548664 1684 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:17.549661 1684 webserver.cc:460] Webserver started at http://127.20.88.62:40137/ using document root <none> and password file <none>
I20260928 23:57:17.550298 1684 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:17.550405 1684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:17.550654 1684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:17.553076 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "edfe8cead97749daa052b37d04eb9419"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "aabd69e252355c9ef9eda4e2dbf8b814"
server_key_iv: "64a7be3408357cf7b47a85b56b8beb7d"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.553719 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "edfe8cead97749daa052b37d04eb9419"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "aabd69e252355c9ef9eda4e2dbf8b814"
server_key_iv: "64a7be3408357cf7b47a85b56b8beb7d"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.557855 1684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:57:17.560734 1699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:17.562140 1684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:57:17.562322 1684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "edfe8cead97749daa052b37d04eb9419"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "aabd69e252355c9ef9eda4e2dbf8b814"
server_key_iv: "64a7be3408357cf7b47a85b56b8beb7d"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.562465 1684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:17.594738 1684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:17.598533 1684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:17.598788 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:17.606978 1684 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:37187
I20260928 23:57:17.606993 1751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:37187 every 8 connection(s)
I20260928 23:57:17.608057 1684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:17.611094 1752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:17.612507 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1684
I20260928 23:57:17.612609 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:57:17.612903 20832 external_mini_cluster.cc:1428] Setting key 809743c8781f76b4d3c78ec8f1d2923e
I20260928 23:57:17.615382 1752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.88.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.61@KRBTEST.COM" created.
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:17.634966 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.619551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.20.88.62:37187)
0928 23:57:17.619929 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:17.619942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:17.619994 (+ 52us) server_negotiation.cc:407] Connection header received
0928 23:57:17.620635 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:17.620642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:17.620938 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:17.621182 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:17.622174 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.623277 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.624589 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.625136 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.625286 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.628694 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:17.628704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:17.628724 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:17.628760 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:17.631253 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.631928 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.631936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.631938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.632112 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.632600 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.632604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.632606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.633256 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:17.633415 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:17.633821 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 23:57:17.634177 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260928 23:57:17.634966 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.618501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37187 (local address 127.0.0.1:36836)
0928 23:57:17.619096 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.619159 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:17.620208 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:17.621214 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:17.621219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:17.621683 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:17.621978 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.621995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.623436 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.623439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.624430 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.624441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.625325 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.625332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.625569 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.626296 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:17.626325 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:17.628503 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:17.631437 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.631447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.631458 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.631771 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.632316 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.632319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.632321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.632467 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.633389 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:17.633394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:17.633530 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 23:57:17.633682 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 23:57:17.634064 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":45,"thread_start_us":186,"threads_started":1}
I20260928 23:57:17.637831 1752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:17.638727 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.638437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51626)
0928 23:57:17.638576 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.638657 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 23:57:17.639081 1702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
I20260928 23:57:17.639958 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.639746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51636)
0928 23:57:17.639850 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.639895 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:57:17.640190 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:32893: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
I20260928 23:57:17.643941 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:32893
--webserver_interface=127.20.88.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:17.689559 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } attempt: 1
I20260928 23:57:17.690481 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.690163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51646)
0928 23:57:17.690334 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.690404 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 23:57:17.691323 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.691101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51654)
0928 23:57:17.691206 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.691252 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:57:17.691566 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:32893: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
I20260928 23:57:17.759832 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } attempt: 2
I20260928 23:57:17.760828 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.760509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51658)
0928 23:57:17.760681 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.760747 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 23:57:17.761821 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.761515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51666)
0928 23:57:17.761666 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.761734 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 23:57:17.762028 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:32893: Network error: Client connection negotiation failed: client connection to 127.20.88.61:32893: connect: Connection refused (error 111)
W20260928 23:57:17.766358 1759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:17.766711 1759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:17.766814 1759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:17.771291 1759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:17.771399 1759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:17.771437 1759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:17.771497 1759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:17.771549 1759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:17.777647 1759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.88.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:17.778895 1759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:17.780128 1759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:17.789198 1765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:17.789162 1764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:17.789179 1767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:17.789434 1759 server_base.cc:1034] running on GCE node
I20260928 23:57:17.790066 1759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:17.791359 1759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:17.792557 1759 hybrid_clock.cc:648] HybridClock initialized: now 1790639837792522 us; error 46 us; skew 500 ppm
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:17.795583 1759 init.cc:377] Logged in from keytab as kudu/127.20.88.61@KRBTEST.COM (short username kudu)
I20260928 23:57:17.796658 1759 webserver.cc:460] Webserver started at http://127.20.88.61:35645/ using document root <none> and password file <none>
I20260928 23:57:17.797272 1759 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:17.797341 1759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:17.797602 1759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:17.800201 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data/instance:
uuid: "f90a87bbd4f1452eb60cfac6671baddf"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "7950f09e4b84e9b43829d74be78aa292"
server_key_iv: "3109452fa51e581e63e28d67785244f4"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.800849 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal/instance:
uuid: "f90a87bbd4f1452eb60cfac6671baddf"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "7950f09e4b84e9b43829d74be78aa292"
server_key_iv: "3109452fa51e581e63e28d67785244f4"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.804625 1759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:17.807126 1774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:17.808528 1759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:17.808710 1759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "f90a87bbd4f1452eb60cfac6671baddf"
format_stamp: "Formatted at 2026-09-28 23:57:17 on dist-test-slave-wvpw"
server_key: "7950f09e4b84e9b43829d74be78aa292"
server_key_iv: "3109452fa51e581e63e28d67785244f4"
server_key_version: "encryptionkey@0"
I20260928 23:57:17.808846 1759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:17.826915 1759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:17.830476 1759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:17.830724 1759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:17.838795 1759 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.61:32893
I20260928 23:57:17.838820 1826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.61:32893 every 8 connection(s)
I20260928 23:57:17.839816 1759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
I20260928 23:57:17.842016 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1759
I20260928 23:57:17.842123 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-1/wal/instance
I20260928 23:57:17.842402 20832 external_mini_cluster.cc:1428] Setting key 537adab461aec39e1203fd61cda088b8
I20260928 23:57:17.842847 1827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:17.846838 1827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.88.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.60@KRBTEST.COM" created.
I20260928 23:57:17.853235 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } attempt: 3
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:57:17.864104 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.848919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.20.88.62:37187)
0928 23:57:17.849057 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:17.849061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:17.850130 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 23:57:17.851258 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:17.851262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:17.851331 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:17.851439 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:17.852733 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.853333 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.854784 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.855167 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.855285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.859279 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:17.859283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:17.859286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:17.859325 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:17.861415 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.862265 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.862269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.862272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.862388 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.862793 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.862800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.862803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.863012 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:17.863124 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:17.863797 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 23:57:17.863933 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:57:17.865271 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.853761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51682)
0928 23:57:17.853997 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.854015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:17.854094 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:17.854907 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:17.854910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:17.855105 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:17.855228 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.855236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.856459 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.856465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.857036 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.857045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.857864 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.857870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.858020 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.858584 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:17.858601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:17.860297 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:17.863606 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.863611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.863613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.863819 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.864147 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.864150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.864152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.864267 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.864774 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:17.864778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:17.864878 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 23:57:17.864996 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 23:57:17.865132 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 23:57:17.865329 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.849429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37187 (local address 127.0.0.1:36842)
0928 23:57:17.849968 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.850011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:17.851037 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:17.851622 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:17.851630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:17.851970 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:17.852577 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.852592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.853489 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.853496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.854639 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.854651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.855396 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.855403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.855771 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.857262 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:17.857282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:17.859119 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:17.861613 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.861630 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.861654 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.862105 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.862506 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.862509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.862511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.862677 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.863135 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:17.863144 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:17.863517 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 23:57:17.863723 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 23:57:17.864163 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":82,"thread_start_us":160,"threads_started":1}
I20260928 23:57:17.865353 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.853888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.20.88.61:32893)
0928 23:57:17.854364 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:17.854378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:17.854405 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:57:17.854451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:17.854459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:17.854678 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:17.854915 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:17.855383 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.856353 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.857164 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.857694 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.857864 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.861602 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:17.861610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:17.861627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:17.861658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:17.863489 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.863936 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.863942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.863944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.864059 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.864383 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.864388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.864391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.864658 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:17.864795 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:17.865060 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:57:17.865185 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":362,"thread_start_us":94,"threads_started":1}
I20260928 23:57:17.867157 1827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:17.867751 1752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:17.868495 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.868205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54142)
0928 23:57:17.868366 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.868429 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 23:57:17.869367 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.869087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54148)
0928 23:57:17.869238 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.869300 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 23:57:17.869508 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
I20260928 23:57:17.871861 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:41949
--webserver_interface=127.20.88.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:57:17 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:57:17.877274 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.867820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51698)
0928 23:57:17.867939 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.867954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:17.868036 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:17.868447 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:17.868449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:17.868622 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:17.868739 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.868747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.869741 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.869744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.870371 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:17.870380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.870897 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.870900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:17.870977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.871527 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:17.871548 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:17.872986 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:17.875002 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.875008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.875011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.875204 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.875863 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:17.875868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:17.875871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:17.875976 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:17.876567 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:17.876570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:17.876681 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 23:57:17.876822 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 23:57:17.877002 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 23:57:17.877321 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.867857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.20.88.61:32893)
0928 23:57:17.867982 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:17.867985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:17.867999 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:57:17.868174 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:17.868177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:17.868224 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:17.868336 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:17.868898 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.869616 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.870479 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:17.870705 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:17.870904 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:17.873145 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:17.873151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:17.873155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:17.873184 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:17.874869 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.875303 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.875308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.875310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.875406 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:17.876094 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:17.876100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:17.876105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:17.876415 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:17.876527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:17.876957 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 23:57:17.877155 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:57:17.878429 1827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:17.879289 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.878928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54160)
0928 23:57:17.879084 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.879193 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 23:57:17.879911 1775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
I20260928 23:57:17.880999 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.880681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54166)
0928 23:57:17.880837 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.880910 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:57:17.881192 1827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
I20260928 23:57:17.900822 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 1
I20260928 23:57:17.901715 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.901400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54176)
0928 23:57:17.901560 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.901632 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 23:57:17.902443 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.902220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54180)
0928 23:57:17.902327 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.902386 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:57:17.902580 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
I20260928 23:57:17.930546 1827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 1
I20260928 23:57:17.931519 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.931162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54186)
0928 23:57:17.931345 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.931421 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 23:57:17.932626 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.932293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54202)
0928 23:57:17.932468 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.932538 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:57:17.932893 1827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
I20260928 23:57:17.977782 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 2
I20260928 23:57:17.978741 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.978373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54206)
0928 23:57:17.978562 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.978643 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 23:57:17.979552 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:17.979311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54212)
0928 23:57:17.979429 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:17.979485 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:57:17.979699 1752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
W20260928 23:57:17.992787 1833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:17.993103 1833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:17.993147 1833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:17.997622 1833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:17.997706 1833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:17.997728 1833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:17.997746 1833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:17.997762 1833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:18.001082 1827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 2
I20260928 23:57:18.002003 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.001679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54226)
0928 23:57:18.001854 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.001922 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 23:57:18.002826 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.002584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54242)
0928 23:57:18.002723 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.002772 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 23:57:18.003051 1827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:41949: Network error: Client connection negotiation failed: client connection to 127.20.88.60:41949: connect: Connection refused (error 111)
I20260928 23:57:18.004410 1833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37187,127.20.88.61:32893,127.20.88.60:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.88.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:18.005558 1833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:18.006642 1833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:18.016304 1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:18.016304 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:18.016350 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:18.016755 1833 server_base.cc:1034] running on GCE node
I20260928 23:57:18.017409 1833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:18.018673 1833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:18.019878 1833 hybrid_clock.cc:648] HybridClock initialized: now 1790639838019851 us; error 42 us; skew 500 ppm
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:18.023180 1833 init.cc:377] Logged in from keytab as kudu/127.20.88.60@KRBTEST.COM (short username kudu)
I20260928 23:57:18.024313 1833 webserver.cc:460] Webserver started at http://127.20.88.60:36637/ using document root <none> and password file <none>
I20260928 23:57:18.024886 1833 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:18.024948 1833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:18.025133 1833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:18.027667 1833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data/instance:
uuid: "09a0e4b88be34eeb9905310b15e7f619"
format_stamp: "Formatted at 2026-09-28 23:57:18 on dist-test-slave-wvpw"
server_key: "592d2ad18726486ad014d99f6017db8b"
server_key_iv: "ad202eec242a55a59858282faaa1374b"
server_key_version: "encryptionkey@0"
I20260928 23:57:18.028319 1833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal/instance:
uuid: "09a0e4b88be34eeb9905310b15e7f619"
format_stamp: "Formatted at 2026-09-28 23:57:18 on dist-test-slave-wvpw"
server_key: "592d2ad18726486ad014d99f6017db8b"
server_key_iv: "ad202eec242a55a59858282faaa1374b"
server_key_version: "encryptionkey@0"
I20260928 23:57:18.032688 1833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:57:18.035604 1849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:18.037343 1833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:18.037515 1833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "09a0e4b88be34eeb9905310b15e7f619"
format_stamp: "Formatted at 2026-09-28 23:57:18 on dist-test-slave-wvpw"
server_key: "592d2ad18726486ad014d99f6017db8b"
server_key_iv: "ad202eec242a55a59858282faaa1374b"
server_key_version: "encryptionkey@0"
I20260928 23:57:18.037691 1833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:18.058220 1833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:18.061868 1833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:18.062101 1833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:18.070007 1833 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.60:41949
I20260928 23:57:18.070079 1901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.60:41949 every 8 connection(s)
I20260928 23:57:18.071097 1833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
I20260928 23:57:18.074280 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:18.078361 1902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:18.078855 1752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 3
I20260928 23:57:18.080073 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1833
I20260928 23:57:18.080174 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639669102918-20832-0/minicluster-data/master-2/wal/instance
I20260928 23:57:18.080493 20832 external_mini_cluster.cc:1428] Setting key 730700fbad0c6240fa3ef3b54a3df1a1
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:57:18.093976 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.079412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54252)
0928 23:57:18.079578 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.079597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:18.079713 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:18.082594 (+ 2881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:18.082598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:18.082801 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:18.082912 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.082922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.084883 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.084887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.085671 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.085683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.086519 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.086523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.086620 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.087111 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:18.087130 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:18.088583 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:18.091606 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.091610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.091612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.091947 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.092485 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.092492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.092589 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.093437 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:18.093439 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:18.093532 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:57:18.093627 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:57:18.093795 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1,"spinlock_wait_cycles":1792}
I20260928 23:57:18.094157 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.081138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.20.88.62:37187)
0928 23:57:18.081271 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.081276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.081814 (+ 538us) server_negotiation.cc:407] Connection header received
0928 23:57:18.082271 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.082274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.082354 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.082411 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.083549 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.084046 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.085336 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.085551 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.085674 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.089511 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.089514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.089516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.089539 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.091672 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092555 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.092561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.092566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.092673 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092985 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.092988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.092990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.093264 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.093357 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.093848 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.093991 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":6272}
I20260928 23:57:18.094519 1827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } attempt: 3
I20260928 23:57:18.095288 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.080281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.20.88.60:41949)
0928 23:57:18.080815 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.080839 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.080889 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:57:18.081892 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.081912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.082212 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.082443 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.083068 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.084745 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.085810 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.086378 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.086501 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.088770 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.088776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.088790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.088840 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.091433 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092080 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.092086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.092088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.092360 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092692 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.092695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.092697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.093321 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.094563 (+ 1242us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.094703 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.094846 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":353,"spinlock_wait_cycles":4608,"thread_start_us":162,"threads_started":1}
I20260928 23:57:18.095289 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.081079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37187 (local address 127.0.0.1:36844)
0928 23:57:18.081659 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.081708 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:18.082020 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:18.082654 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:18.082660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:18.083155 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:18.083411 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.083423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.084169 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.084176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.085208 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.085219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.085719 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.085724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.085921 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.086960 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:18.086982 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:18.089304 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:18.092038 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092050 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.092064 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.092450 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.092787 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.092790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.092791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.092869 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.093423 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:18.093429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:18.093568 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 23:57:18.093758 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 23:57:18.094111 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":112,"threads_started":1}
I20260928 23:57:18.096437 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.081847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.20.88.60:41949)
0928 23:57:18.082106 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.082119 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.082133 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:57:18.082165 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.082167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.082211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.082443 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.083309 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.084745 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.086403 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.086671 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.086795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.089878 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.089881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.089882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.089904 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.092809 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.093266 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.093271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.093273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.093366 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.093652 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.093655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.093657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.093885 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.095056 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.095138 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.095261 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":920,"server-negotiator.queue_time_us":195,"spinlock_wait_cycles":7680,"thread_start_us":103,"threads_started":1}
I20260928 23:57:18.097563 1902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:18.100430 1752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: Bootstrap starting.
I20260928 23:57:18.103590 1752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: Neither blocks nor log segments found. Creating new log.
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:57:18.104789 1752 log.cc:824] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:57:18.107332 1752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: No bootstrap required, opened a new log
I20260928 23:57:18.109547 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.095093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54276)
0928 23:57:18.095428 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.095448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:18.095550 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:18.096054 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:18.096057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:18.096217 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:18.096387 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.096393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.100932 (+ 4539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.100936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.101731 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.101746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.102371 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.102375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.102457 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.103210 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:18.103226 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:18.105393 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:18.107322 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.107325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.107327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.107907 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.108275 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.108279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.108281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.108380 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.108901 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:18.108906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:18.109025 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 23:57:18.109187 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:57:18.109357 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260928 23:57:18.110555 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.095702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.20.88.60:41949)
0928 23:57:18.095828 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.095832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.095845 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:57:18.095877 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.095879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.095919 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.096000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.096893 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.100798 (+ 3905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.101897 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.102237 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.102713 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.105556 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.105560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.105566 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.105599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.107191 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.108032 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.108037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.108038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.108129 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.108503 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.108509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.108514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.108773 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.110185 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.110265 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.110391 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:57:18.110757 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.098390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.20.88.61:32893)
0928 23:57:18.100405 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.100409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.100421 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:57:18.100457 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.100459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.100496 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.100591 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.101389 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.102137 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.103841 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.104279 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.104359 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.106939 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.106944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.106947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.106978 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.108676 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.109152 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.109156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.109157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.109240 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.109531 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.109534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.109536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.110045 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.110134 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.110470 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.110591 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1917}
I20260928 23:57:18.110857 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.098823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.20.88.62:37187)
0928 23:57:18.098958 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.098963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.098978 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:57:18.099020 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.099024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.099070 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.099166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.099730 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.100480 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.101771 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.102099 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.102220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.105687 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.105693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.105696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.105726 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.108470 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.108995 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.108998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.109000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.109089 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.109413 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.109416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.109418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.109632 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.109830 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.110573 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.110717 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:57:18.111830 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.098101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:32893 (local address 127.0.0.1:51714)
0928 23:57:18.098393 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.098409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:18.098484 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:18.100934 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:18.100936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:18.101128 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:18.101248 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.101255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.102271 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.102274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.103658 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.103673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.104520 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.104527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.104624 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.105147 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:18.105163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:18.106766 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:18.108801 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.108804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.108806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.109039 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.109345 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.109348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.109351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.109432 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.110170 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:18.110173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:18.110277 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:57:18.111503 (+ 1226us) client_negotiation.cc:240] Negotiation successful
0928 23:57:18.111665 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260928 23:57:18.112030 1752 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.112437 1752 raft_consensus.cc:374] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:18.112521 1752 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfe8cead97749daa052b37d04eb9419, State: Initialized, Role: FOLLOWER
I20260928 23:57:18.113106 1752 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.113696 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.113917 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.114398 1827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: Bootstrap starting.
I20260928 23:57:18.114656 1752 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:18.117048 1902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:57:18.118018 1827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: Neither blocks nor log segments found. Creating new log.
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:57:18.118988 1827 log.cc:824] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:18.121390 1827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: No bootstrap required, opened a new log
I20260928 23:57:18.120289 1684 master_runner.cc:186] Error getting master registration for 127.20.88.61:32893: OK, instance_id { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" instance_seqno: 1790639837836577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:57:18.125301 1684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 23:57:18 dist-test-slave-wvpw krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:57:18.130290 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.112222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.20.88.61:32893)
0928 23:57:18.112374 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.112378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.112526 (+ 148us) server_negotiation.cc:407] Connection header received
0928 23:57:18.112685 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.112688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.112737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.112820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.113688 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.114377 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.115461 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.115819 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.115926 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.124907 (+ 8981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.124912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.124914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.124949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.127596 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.128170 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.128174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.128176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.128501 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.128869 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.128872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.128874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.129108 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.129371 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.129686 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.130094 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:57:18.132833 1827 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.133128 1827 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:18.133189 1827 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90a87bbd4f1452eb60cfac6671baddf, State: Initialized, Role: FOLLOWER
I20260928 23:57:18.133808 1827 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.134617 1922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.134796 1922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.135166 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.117588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:41949 (local address 127.0.0.1:54286)
0928 23:57:18.117685 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:18.117700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:18.117787 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:18.119760 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:18.119763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:18.119942 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:18.120066 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.120073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.125360 (+ 5287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.125364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.126448 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:18.126461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.127378 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.127381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:18.127467 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.128548 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:18.128570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:18.130417 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:18.133531 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.133534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.133537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.133770 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.134217 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:18.134220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:18.134222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:18.134303 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:18.134720 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:18.134722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:18.134795 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 23:57:18.134867 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:57:18.135008 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:57:18.135973 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.117737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.20.88.60:41949)
0928 23:57:18.119500 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.119505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.119522 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:57:18.119566 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.119572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.119621 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.124341 (+ 4720us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.124499 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.125228 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.126595 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.127251 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.127712 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.130558 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.130562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.130564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.130586 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.133414 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.133883 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.133886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.133888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.134115 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.134410 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.134412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.134414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.134626 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.135481 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.135587 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.135835 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677}
I20260928 23:57:18.136004 1827 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:18.137147 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:57:18.139701 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: Bootstrap starting.
I20260928 23:57:18.142943 1902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:18.143859 1902 log.cc:824] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: Log is configured to *not* fsync() on all Append() calls
W20260928 23:57:18.143450 1920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:57:18.144780 1920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:57:18.154601 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.140309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.20.88.62:37187)
0928 23:57:18.140440 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.140445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.143960 (+ 3515us) server_negotiation.cc:407] Connection header received
0928 23:57:18.145044 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.145048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.145102 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.145180 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.146232 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.146978 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.148152 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.148638 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.148738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.150684 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.150690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.150693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.150722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.152627 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.153121 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.153125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.153127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.153235 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.153528 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.153532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.153534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.153780 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.153897 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.154293 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.154432 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:57:18.154613 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: No bootstrap required, opened a new log
W20260928 23:57:18.158129 1933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:57:18.158253 1933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:57:18.153585 1759 master_runner.cc:186] Error getting master registration for 127.20.88.60:41949: OK, instance_id { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" instance_seqno: 1790639838067728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:57:18.159283 1759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:57:18.159113 1902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.159402 1902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:18.159471 1902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0e4b88be34eeb9905310b15e7f619, State: Initialized, Role: FOLLOWER
I20260928 23:57:18.160099 1902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.166076 1902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:18.168057 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.140602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.20.88.60:41949)
0928 23:57:18.140716 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.140721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.148813 (+ 8092us) server_negotiation.cc:407] Connection header received
0928 23:57:18.149006 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.149010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.149068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.149122 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.151615 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.152527 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.157041 (+ 4514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.157404 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.157513 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.162151 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.162156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.162159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.162210 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.165278 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.165835 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.165840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.165842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.165932 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.166389 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.166393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.166394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.166670 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.166791 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.167718 (+ 927us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.167873 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:57:18.170114 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:18.141183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.20.88.61:32893)
0928 23:57:18.141295 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:18.141299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:18.150402 (+ 9103us) server_negotiation.cc:407] Connection header received
0928 23:57:18.150586 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:18.150591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:18.150647 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:18.150710 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:18.157098 (+ 6388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.157820 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.159888 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:18.160328 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:18.160415 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:18.163773 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:18.163777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:18.163780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:18.163815 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:18.167912 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.168586 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.168592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.168594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.168703 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:18.169028 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:18.169031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:18.169033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:18.169282 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:18.169418 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:18.169793 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 23:57:18.169938 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:57:18.175367 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.175545 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:57:18.179344 1955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:57:18.179557 1955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:57:18.182963 1833 master_runner.cc:263] No leader master found from master 09a0e4b88be34eeb9905310b15e7f619
I20260928 23:57:18.183050 1833 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:57:18.251085 1944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:18.251325 1944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.252718 1944 leader_election.cc:288] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfe8cead97749daa052b37d04eb9419 (127.20.88.62:37187), f90a87bbd4f1452eb60cfac6671baddf (127.20.88.61:32893)
I20260928 23:57:18.253495 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09a0e4b88be34eeb9905310b15e7f619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90a87bbd4f1452eb60cfac6671baddf" is_pre_election: true
I20260928 23:57:18.253945 1801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0e4b88be34eeb9905310b15e7f619 in term 0.
I20260928 23:57:18.254559 1852 leader_election.cc:302] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0e4b88be34eeb9905310b15e7f619, f90a87bbd4f1452eb60cfac6671baddf; no voters:
I20260928 23:57:18.254963 1944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:57:18.255046 1944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:57:18.255088 1944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:18.256911 1726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09a0e4b88be34eeb9905310b15e7f619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfe8cead97749daa052b37d04eb9419" is_pre_election: true
I20260928 23:57:18.257232 1726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0e4b88be34eeb9905310b15e7f619 in term 0.
I20260928 23:57:18.260062 1944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.260562 1944 leader_election.cc:288] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [CANDIDATE]: Term 1 election: Requested vote from peers edfe8cead97749daa052b37d04eb9419 (127.20.88.62:37187), f90a87bbd4f1452eb60cfac6671baddf (127.20.88.61:32893)
I20260928 23:57:18.261008 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09a0e4b88be34eeb9905310b15e7f619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90a87bbd4f1452eb60cfac6671baddf"
I20260928 23:57:18.261144 1801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:18.263087 1801 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0e4b88be34eeb9905310b15e7f619 in term 1.
I20260928 23:57:18.263931 1726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09a0e4b88be34eeb9905310b15e7f619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfe8cead97749daa052b37d04eb9419"
I20260928 23:57:18.264060 1726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:18.265412 1726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0e4b88be34eeb9905310b15e7f619 in term 1.
I20260928 23:57:18.263521 1852 leader_election.cc:302] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0e4b88be34eeb9905310b15e7f619, f90a87bbd4f1452eb60cfac6671baddf; no voters:
I20260928 23:57:18.267738 1944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:18.268019 1944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [term 1 LEADER]: Becoming Leader. State: Replica: 09a0e4b88be34eeb9905310b15e7f619, State: Running, Role: LEADER
I20260928 23:57:18.268471 1944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } }
I20260928 23:57:18.270357 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a0e4b88be34eeb9905310b15e7f619. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.270561 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:18.272673 1959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:18.273721 1959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:18.282284 1801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [term 1 FOLLOWER]: Refusing update from remote peer 09a0e4b88be34eeb9905310b15e7f619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:18.284740 1957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:18.290514 1922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a0e4b88be34eeb9905310b15e7f619. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.290725 1922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.292426 1726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [term 1 FOLLOWER]: Refusing update from remote peer 09a0e4b88be34eeb9905310b15e7f619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:18.293367 1957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:18.298478 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a0e4b88be34eeb9905310b15e7f619. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.298664 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.307689 1957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.307865 1957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:18.309538 1922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.309674 1922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.311913 1909 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.312072 1909 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:57:18.312779 1960 mvcc.cc:205] Tried to move back new op lower bound from 7334460777589137408 to 7334460777551298560. Current Snapshot: MvccSnapshot[applied={T|T < 7334460777589137408}]
I20260928 23:57:18.313244 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09a0e4b88be34eeb9905310b15e7f619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfe8cead97749daa052b37d04eb9419" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37187 } } peers { permanent_uuid: "f90a87bbd4f1452eb60cfac6671baddf" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 32893 } } peers { permanent_uuid: "09a0e4b88be34eeb9905310b15e7f619" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 41949 } } }
I20260928 23:57:18.313369 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:18.314092 1961 mvcc.cc:205] Tried to move back new op lower bound from 7334460777589137408 to 7334460777551298560. Current Snapshot: MvccSnapshot[applied={T|T < 7334460777589137408 or (T in {7334460777589137408})}]
I20260928 23:57:18.321448 1959 catalog_manager.cc:1347] Generated new cluster ID: a3fd81bd09c441d68946c8557d197125
I20260928 23:57:18.321539 1959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:18.346444 1959 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:57:18.347950 1959 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:18.375038 1959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09a0e4b88be34eeb9905310b15e7f619: Generated new TSK 0
I20260928 23:57:18.375988 1959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:57:19.146143 1920 catalog_manager.cc:1259] Loaded cluster ID: a3fd81bd09c441d68946c8557d197125
I20260928 23:57:19.146245 1920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: loading cluster ID for follower catalog manager: success
I20260928 23:57:19.147921 1920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: acquiring CA information for follower catalog manager: success
I20260928 23:57:19.149170 1920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edfe8cead97749daa052b37d04eb9419: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:57:19.159627 1933 catalog_manager.cc:1259] Loaded cluster ID: a3fd81bd09c441d68946c8557d197125
I20260928 23:57:19.159714 1933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: loading cluster ID for follower catalog manager: success
I20260928 23:57:19.161382 1933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: acquiring CA information for follower catalog manager: success
I20260928 23:57:19.162549 1933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f90a87bbd4f1452eb60cfac6671baddf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:57:19.447929 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1684
I20260928 23:57:19.455046 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1759
I20260928 23:57:19.461436 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1833
2026-09-28T23:57:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3232 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:57:19 dist-test-slave-wvpw krb5kdc[1975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:57:19 dist-test-slave-wvpw krb5kdc[1975](info): set up 2 sockets
Sep 28 23:57:19 dist-test-slave-wvpw krb5kdc[1975](info): commencing operation
krb5kdc: starting...
W20260928 23:57:20.513182 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:57:20.644061 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44807
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:20.762661 1988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:20.762966 1988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:20.763010 1988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:20.767204 1988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:20.767275 1988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:20.767297 1988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:20.767314 1988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:20.767329 1988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:20.772940 1988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:20.774139 1988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:20.775257 1988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:20.784651 1994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:20.784651 1996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:20.784796 1993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:20.784843 1988 server_base.cc:1034] running on GCE node
I20260928 23:57:20.785743 1988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:20.786962 1988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:20.788146 1988 hybrid_clock.cc:648] HybridClock initialized: now 1790639840788110 us; error 54 us; skew 500 ppm
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639840, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:20.791337 1988 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:20.792495 1988 webserver.cc:460] Webserver started at http://127.20.88.62:40793/ using document root <none> and password file <none>
I20260928 23:57:20.793139 1988 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:20.793246 1988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:20.793494 1988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:20.795835 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "479cfbdeee0044558d92342f6e0a5731"
format_stamp: "Formatted at 2026-09-28 23:57:20 on dist-test-slave-wvpw"
server_key: "7fa0e8a94221914a1df5ed739e2d75dd"
server_key_iv: "f2c3d6c0fb820100a9af2bbdc0914c7a"
server_key_version: "encryptionkey@0"
I20260928 23:57:20.796535 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "479cfbdeee0044558d92342f6e0a5731"
format_stamp: "Formatted at 2026-09-28 23:57:20 on dist-test-slave-wvpw"
server_key: "7fa0e8a94221914a1df5ed739e2d75dd"
server_key_iv: "f2c3d6c0fb820100a9af2bbdc0914c7a"
server_key_version: "encryptionkey@0"
I20260928 23:57:20.800872 1988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:57:20.803787 2003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:20.805497 1988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:20.805682 1988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "479cfbdeee0044558d92342f6e0a5731"
format_stamp: "Formatted at 2026-09-28 23:57:20 on dist-test-slave-wvpw"
server_key: "7fa0e8a94221914a1df5ed739e2d75dd"
server_key_iv: "f2c3d6c0fb820100a9af2bbdc0914c7a"
server_key_version: "encryptionkey@0"
I20260928 23:57:20.805835 1988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:20.821794 1988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:20.825532 1988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:20.825794 1988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:20.833562 1988 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:44807
I20260928 23:57:20.833575 2055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:44807 every 8 connection(s)
I20260928 23:57:20.834638 1988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:20.837708 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:20.841687 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 1988
I20260928 23:57:20.841830 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:57:20.842114 20832 external_mini_cluster.cc:1428] Setting key 558ac283680bbb6037dfc759b4075ff7
I20260928 23:57:20.844406 2056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731: Bootstrap starting.
I20260928 23:57:20.847041 2056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:20.847856 2056 log.cc:824] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:20.850162 2056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731: No bootstrap required, opened a new log
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:20.854449 2056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } }
I20260928 23:57:20.854732 2056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:20.854817 2056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cfbdeee0044558d92342f6e0a5731, State: Initialized, Role: FOLLOWER
I20260928 23:57:20.855432 2056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } }
I20260928 23:57:20.855607 2056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:20.855676 2056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:20.855774 2056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:20.857089 2056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } }
I20260928 23:57:20.857497 2056 leader_election.cc:302] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 479cfbdeee0044558d92342f6e0a5731; no voters:
I20260928 23:57:20.857847 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:20.843681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.20.88.62:44807)
0928 23:57:20.844154 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:20.844170 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:20.844207 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:57:20.845004 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:20.845024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:20.845376 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:20.845703 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:20.846664 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:20.847744 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:20.848848 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:20.849325 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:20.849455 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:20.852316 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:20.852321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:20.852337 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:20.852366 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:20.854210 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:20.854711 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:20.854716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:20.854721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:20.854892 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:20.855243 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:20.855247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:20.855248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:20.855843 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:20.856063 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:20.856388 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 23:57:20.856728 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":147,"threads_started":1}
I20260928 23:57:20.857847 2056 leader_election.cc:288] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:57:20.857977 2061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:20.858197 2061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [term 1 LEADER]: Becoming Leader. State: Replica: 479cfbdeee0044558d92342f6e0a5731, State: Running, Role: LEADER
I20260928 23:57:20.858506 2061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } }
I20260928 23:57:20.859236 2056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:20.859833 2062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "479cfbdeee0044558d92342f6e0a5731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } } }
I20260928 23:57:20.859956 2062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:20.860186 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 479cfbdeee0044558d92342f6e0a5731. Latest consensus state: current_term: 1 leader_uuid: "479cfbdeee0044558d92342f6e0a5731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479cfbdeee0044558d92342f6e0a5731" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44807 } } }
I20260928 23:57:20.860321 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:20.860991 2066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:20.861084 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:57:20.864941 2066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:20.871412 2066 catalog_manager.cc:1347] Generated new cluster ID: c29dccd471824eb29f51bd8a03b5f90e
I20260928 23:57:20.871528 2066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:20.894008 2066 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:57:20.895143 2066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:20.903225 2066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 479cfbdeee0044558d92342f6e0a5731: Generated new TSK 0
I20260928 23:57:20.904054 2066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:57:20.917295 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 1988
2026-09-28T23:57:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1453 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[2083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[2083](info): set up 2 sockets
Sep 28 23:57:20 dist-test-slave-wvpw krb5kdc[2083](info): commencing operation
krb5kdc: starting...
W20260928 23:57:21.961068 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:57:22.096014 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:22.214310 2096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:22.214680 2096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:22.214766 2096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:22.219029 2096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:22.219163 2096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:22.219202 2096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:22.219239 2096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:22.219280 2096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:22.225044 2096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:22.226234 2096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:22.227310 2096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:22.235988 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:22.236039 2096 server_base.cc:1034] running on GCE node
W20260928 23:57:22.236061 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.236268 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:22.236866 2096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:22.238096 2096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:22.239320 2096 hybrid_clock.cc:648] HybridClock initialized: now 1790639842239307 us; error 31 us; skew 500 ppm
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:22.249577 2096 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:22.250684 2096 webserver.cc:460] Webserver started at http://127.20.88.62:35067/ using document root <none> and password file <none>
I20260928 23:57:22.251307 2096 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:22.251374 2096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:22.251559 2096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:22.253958 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "07aa4c60080141ef9e9264c5d70816d7"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "88814a16cdd09cd9a391e3f0598487cc"
server_key_iv: "4da40041b4877f9f5f938abab0459edf"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.254551 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "07aa4c60080141ef9e9264c5d70816d7"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "88814a16cdd09cd9a391e3f0598487cc"
server_key_iv: "4da40041b4877f9f5f938abab0459edf"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.258767 2096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:57:22.261723 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.263221 2096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:57:22.263348 2096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "07aa4c60080141ef9e9264c5d70816d7"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "88814a16cdd09cd9a391e3f0598487cc"
server_key_iv: "4da40041b4877f9f5f938abab0459edf"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.263461 2096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:22.276191 2096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:22.279742 2096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:22.279907 2096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:22.287508 2096 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:46439
I20260928 23:57:22.287837 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:46439 every 8 connection(s)
I20260928 23:57:22.288762 2096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:22.291744 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:22.293362 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2096
I20260928 23:57:22.293491 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:57:22.293798 20832 external_mini_cluster.cc:1428] Setting key a2ab603ce7fab6f389bbc9da73aeade6
I20260928 23:57:22.298834 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap starting.
I20260928 23:57:22.301765 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:22.302821 2164 log.cc:824] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:22.305406 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: No bootstrap required, opened a new log
I20260928 23:57:22.309486 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:22.295256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.20.88.62:46439)
0928 23:57:22.295724 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:22.295745 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:22.295795 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:57:22.296532 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:22.296553 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:22.296911 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:22.297221 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:22.298165 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.299282 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.300527 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.301047 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.301191 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:22.304473 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:22.304480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:22.304496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:22.304525 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:22.306375 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.306911 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.306916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.306921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.307053 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.307363 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.307365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.307366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.307787 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:22.307936 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:22.308331 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 23:57:22.308679 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":163,"threads_started":1}
I20260928 23:57:22.309934 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:22.310241 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:22.310302 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Initialized, Role: FOLLOWER
I20260928 23:57:22.310807 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:22.310950 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:22.311038 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:22.311175 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:22.312436 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:22.312829 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07aa4c60080141ef9e9264c5d70816d7; no voters:
I20260928 23:57:22.313169 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:57:22.313345 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:22.313608 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 LEADER]: Becoming Leader. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Running, Role: LEADER
I20260928 23:57:22.313967 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:22.314503 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:22.315716 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:22.315860 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:22.316187 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07aa4c60080141ef9e9264c5d70816d7. Latest consensus state: current_term: 1 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:22.316309 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:22.316339 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:22.319954 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:22.326181 2178 catalog_manager.cc:1347] Generated new cluster ID: bebf4e8be7384ebf9d789c7710bb457c
I20260928 23:57:22.326282 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:22.343490 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:57:22.344982 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:22.352749 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Generated new TSK 0
I20260928 23:57:22.353937 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:57:22.387784 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:22.507519 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:22.507838 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:22.507885 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:22.512214 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:22.512328 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:22.512393 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:22.518332 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:22.519438 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:22.520476 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:22.530740 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.530786 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.530786 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:22.531003 2190 server_base.cc:1034] running on GCE node
I20260928 23:57:22.531520 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:22.532347 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:22.533541 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790639842533524 us; error 39 us; skew 500 ppm
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:22.536780 2190 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:22.537837 2190 webserver.cc:460] Webserver started at http://127.20.88.1:42255/ using document root <none> and password file <none>
I20260928 23:57:22.538431 2190 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:22.538504 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:22.538697 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:22.541055 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "a4d14c6334cd1b3b6fd44a11e68cf1c5"
server_key_iv: "163a2000bd572106234580c0dcadfd18"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.541723 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "a4d14c6334cd1b3b6fd44a11e68cf1c5"
server_key_iv: "163a2000bd572106234580c0dcadfd18"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.546198 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:57:22.549196 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.550712 2190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:57:22.550868 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "a4d14c6334cd1b3b6fd44a11e68cf1c5"
server_key_iv: "163a2000bd572106234580c0dcadfd18"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.551010 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:22.565845 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:22.570108 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:22.570361 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:22.571033 2190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:22.571946 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:22.571995 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.572064 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:22.572098 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.581907 2190 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35115
I20260928 23:57:22.581928 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35115 every 8 connection(s)
I20260928 23:57:22.582937 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:22.585071 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2190
I20260928 23:57:22.585204 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:57:22.585522 20832 external_mini_cluster.cc:1428] Setting key 8efb66491ee7311145fe603bcca6dbef
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:57:22.600714 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:22.585233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:54889 (local address 127.20.88.62:46439)
0928 23:57:22.585394 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:22.585398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:22.586597 (+ 1199us) server_negotiation.cc:407] Connection header received
0928 23:57:22.588477 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:22.588483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:22.588554 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:22.588654 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:22.590042 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.590602 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.592514 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.592855 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.592958 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:22.596410 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:22.596414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:22.596416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:22.596446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:22.598122 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.598847 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.598850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.598852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.598941 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.599361 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.599367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.599376 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.599587 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:22.599714 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:22.600352 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 23:57:22.600514 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:57:22.601637 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:22.585693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:54889)
0928 23:57:22.586376 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:22.586465 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:22.588141 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:22.588842 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:22.588851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:22.589318 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:22.589859 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:22.589875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.590768 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.590774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:22.592361 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:22.592379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.593056 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.593063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:22.593334 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:22.594117 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:22.594149 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:22.596101 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:22.598303 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:22.598314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:22.598337 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:22.598704 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:22.599062 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:22.599065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:22.599067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:22.599245 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:22.599712 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:22.599721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:22.600027 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 23:57:22.600273 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 23:57:22.600666 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":183,"threads_started":1}
I20260928 23:57:22.603003 2319 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
I20260928 23:57:22.603287 2319 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:22.603842 2319 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:22.605623 2128 ts_manager.cc:194] Registered new tserver with Master: 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115)
I20260928 23:57:22.607409 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:54889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:57:22.620427 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:22.735721 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:22.736013 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:22.736059 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:22.740351 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:22.740423 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:22.740486 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:22.746495 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:22.747654 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:22.748728 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:22.758800 2329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.758800 2332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.758803 2330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:22.759019 2324 server_base.cc:1034] running on GCE node
I20260928 23:57:22.759598 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:22.760315 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:22.761484 2324 hybrid_clock.cc:648] HybridClock initialized: now 1790639842761462 us; error 37 us; skew 500 ppm
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:22.764532 2324 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:22.765491 2324 webserver.cc:460] Webserver started at http://127.20.88.2:44119/ using document root <none> and password file <none>
I20260928 23:57:22.766078 2324 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:22.766175 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:22.766409 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:22.768680 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "539b4738f4994589b91c6ffca17107ea"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "4c87162e502def35a3a44b852ac7fad6"
server_key_iv: "33ffee6d625563e0545114a253609954"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.769265 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "539b4738f4994589b91c6ffca17107ea"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "4c87162e502def35a3a44b852ac7fad6"
server_key_iv: "33ffee6d625563e0545114a253609954"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.773188 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:57:22.775846 2339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.777240 2324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:22.777390 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "539b4738f4994589b91c6ffca17107ea"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "4c87162e502def35a3a44b852ac7fad6"
server_key_iv: "33ffee6d625563e0545114a253609954"
server_key_version: "encryptionkey@0"
I20260928 23:57:22.777525 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:22.791194 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:22.794816 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:22.795039 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:22.795698 2324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:22.796665 2324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:22.796738 2324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.796811 2324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:22.796854 2324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:22.808012 2324 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:41875
I20260928 23:57:22.808028 2452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:41875 every 8 connection(s)
I20260928 23:57:22.809254 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:22.817658 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2324
I20260928 23:57:22.817767 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:57:22.818049 20832 external_mini_cluster.cc:1428] Setting key 66ad3c047a07c51f898e61af00edd0fc
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:22.826398 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:22.811392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:55591 (local address 127.20.88.62:46439)
0928 23:57:22.811540 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:22.811544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:22.812622 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 23:57:22.814102 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:22.814105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:22.814168 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:22.814250 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:22.815637 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.816399 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.817611 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.817947 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.818067 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:22.821901 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:22.821907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:22.821911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:22.821947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:22.823729 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.824355 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.824362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.824367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.824470 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:22.824933 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:22.824939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:22.824942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:22.825228 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:22.825342 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:22.825977 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 23:57:22.826148 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:57:22.827235 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:22.811844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:55591)
0928 23:57:22.812462 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:22.812511 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:22.813858 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:22.814439 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:22.814447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:22.814909 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:22.815447 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:22.815462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.816548 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.816550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:22.817478 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:22.817487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:22.818114 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:22.818118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:22.818306 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:22.819457 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:22.819480 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:22.821670 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:22.823872 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:22.823879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:22.823895 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:22.824205 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:22.824638 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:22.824646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:22.824648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:22.824813 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:22.825350 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:22.825356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:22.825686 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 23:57:22.825865 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 23:57:22.826170 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
I20260928 23:57:22.828691 2453 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:57:22.828985 2453 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:22.829535 2453 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:22.830722 2128 ts_manager.cc:194] Registered new tserver with Master: 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875)
I20260928 23:57:22.831313 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:55591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:57:22.851049 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:22.969877 2458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:22.970156 2458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:22.970201 2458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:22.974431 2458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:22.974509 2458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:22.974570 2458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:22.980357 2458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:22.981521 2458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:22.982607 2458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:22.992813 2464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.992761 2466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:22.992765 2463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:22.993088 2458 server_base.cc:1034] running on GCE node
I20260928 23:57:22.993661 2458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:22.994397 2458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:22.995591 2458 hybrid_clock.cc:648] HybridClock initialized: now 1790639842995553 us; error 53 us; skew 500 ppm
Sep 28 23:57:22 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:22.998366 2458 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:22.999390 2458 webserver.cc:460] Webserver started at http://127.20.88.3:35517/ using document root <none> and password file <none>
I20260928 23:57:22.999990 2458 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:23.000083 2458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:23.000385 2458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:23.002521 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "864e8b94de4f47b280f2b13e1a947e09"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "8801ac5e9c8281ef2dcd4f8e18874cec"
server_key_iv: "3411e2126b56daea44954ec422445c35"
server_key_version: "encryptionkey@0"
I20260928 23:57:23.003132 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "864e8b94de4f47b280f2b13e1a947e09"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "8801ac5e9c8281ef2dcd4f8e18874cec"
server_key_iv: "3411e2126b56daea44954ec422445c35"
server_key_version: "encryptionkey@0"
I20260928 23:57:23.007022 2458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:57:23.009598 2473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:23.010872 2458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:57:23.011013 2458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "864e8b94de4f47b280f2b13e1a947e09"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "8801ac5e9c8281ef2dcd4f8e18874cec"
server_key_iv: "3411e2126b56daea44954ec422445c35"
server_key_version: "encryptionkey@0"
I20260928 23:57:23.011134 2458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:23.030051 2458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:23.033700 2458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:23.033921 2458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:23.034559 2458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:23.035449 2458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:23.035521 2458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:23.035595 2458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:23.035646 2458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:23.045389 2458 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:34823
I20260928 23:57:23.045408 2586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:34823 every 8 connection(s)
I20260928 23:57:23.046386 2458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:23.048631 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2458
I20260928 23:57:23.048802 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:57:23.049089 20832 external_mini_cluster.cc:1428] Setting key a22b8674b6a8abc507e765a432ad66c6
Sep 28 23:57:23 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639842, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:23.061379 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.048447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:50341 (local address 127.20.88.62:46439)
0928 23:57:23.048603 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.048607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.049618 (+ 1011us) server_negotiation.cc:407] Connection header received
0928 23:57:23.051178 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.051183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.051239 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.051319 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:23.052833 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.053397 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.054617 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.054908 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.055033 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.057858 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:23.057862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:23.057864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:23.057894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:23.059263 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.059825 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.059827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.059828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.059908 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.060302 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.060306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.060307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.060515 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:23.060595 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.061126 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.061245 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:57:23.062475 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.048960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:50341)
0928 23:57:23.049457 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.049514 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.050952 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.051500 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.051507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.051951 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:23.052686 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.052705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.053538 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.053542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.054479 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.054489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.055083 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.055087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.055303 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.055994 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:23.056012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:23.057656 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:23.059382 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.059388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.059403 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.059719 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.059986 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.059989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.059990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.060158 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.060608 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:23.060614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:23.060896 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.061062 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.061421 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":152,"threads_started":1}
I20260928 23:57:23.063712 2587 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
I20260928 23:57:23.063941 2587 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:23.064492 2587 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:23.065523 2128 ts_manager.cc:194] Registered new tserver with Master: 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:23.066118 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:50341
I20260928 23:57:23.075706 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:57:23.086495 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.078200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.20.88.62:46439)
0928 23:57:23.078374 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.078379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.078492 (+ 113us) server_negotiation.cc:407] Connection header received
0928 23:57:23.078663 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.078666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.078724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.078819 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:23.079615 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.080148 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.081122 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.081362 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.081480 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.083090 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:23.083093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:23.083095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:23.083124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:23.084654 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.085123 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.085127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.085128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.085215 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.085502 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.085505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.085507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.085788 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:23.085938 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.086180 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.086312 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 23:57:23.089869 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:57:23.093252 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:57:23.109650 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.103999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34823 (local address 127.0.0.1:55148)
0928 23:57:23.104287 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.104303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.105028 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.105537 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.105539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.105590 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:23.105680 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.105687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106971 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.106974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.108083 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.108093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.109021 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.109025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.109135 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.109176 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.109309 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.109505 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":73,"thread_start_us":100,"threads_started":1}
I20260928 23:57:23.109650 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.104472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.20.88.3:34823)
0928 23:57:23.104983 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.104995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.105016 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:57:23.105191 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.105197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.105392 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.105549 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:23.105805 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106835 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.108223 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.108890 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.109026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.109187 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.109354 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.109465 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":194,"threads_started":1}
I20260928 23:57:23.109694 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.102794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35115 (local address 127.0.0.1:52842)
0928 23:57:23.103442 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.103515 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.103678 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.105003 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.105010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.105029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:23.105288 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.105297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106700 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.106703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.107901 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.107912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.109113 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.109117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.109221 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.109251 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.109299 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.109499 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":33,"thread_start_us":82,"threads_started":1}
I20260928 23:57:23.110330 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.103510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.20.88.2:41875)
0928 23:57:23.103881 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.103910 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.104125 (+ 215us) server_negotiation.cc:407] Connection header received
0928 23:57:23.104314 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.104321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.104842 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.105006 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:23.105422 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106342 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.108336 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.109153 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.109251 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.109387 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.110059 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.110176 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
I20260928 23:57:23.110566 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.102951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.20.88.1:35115)
0928 23:57:23.104461 (+ 1510us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.104468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.104492 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:57:23.104572 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.104579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.104813 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.105241 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:23.105382 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106579 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.108031 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.108959 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.110106 (+ 1147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.110249 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.110301 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.110420 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":110,"threads_started":1}
I20260928 23:57:23.110589 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.102948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41875 (local address 127.0.0.1:55118)
0928 23:57:23.104015 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.104027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.104164 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.105088 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.105091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.105109 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:23.105290 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.105297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.106471 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.106477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.108187 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.108196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.109448 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.109452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.109542 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.109566 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.109634 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.109700 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":166,"thread_start_us":125,"threads_started":1}
I20260928 23:57:23.113098 2521 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29329d64e8446d999d561d199467d1 (DEFAULT_TABLE table=test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:23.113350 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29329d64e8446d999d561d199467d1 (DEFAULT_TABLE table=test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:23.113653 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 4d29329d64e8446d999d561d199467d1 (DEFAULT_TABLE table=test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:23.114377 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d29329d64e8446d999d561d199467d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:23.114375 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d29329d64e8446d999d561d199467d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:23.114984 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d29329d64e8446d999d561d199467d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:23.122044 2604 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Bootstrap starting.
I20260928 23:57:23.122550 2605 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap starting.
I20260928 23:57:23.122974 2606 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Bootstrap starting.
I20260928 23:57:23.125653 2605 tablet_bootstrap.cc:654] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:23.125653 2606 tablet_bootstrap.cc:654] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:23.126637 2604 tablet_bootstrap.cc:654] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:23.126710 2605 log.cc:824] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:23.127223 2606 log.cc:824] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:23.127485 2604 log.cc:824] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:23.129181 2605 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: No bootstrap required, opened a new log
I20260928 23:57:23.129372 2606 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: No bootstrap required, opened a new log
I20260928 23:57:23.129369 2604 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: No bootstrap required, opened a new log
I20260928 23:57:23.129449 2605 ts_tablet_manager.cc:1397] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:57:23.129560 2604 ts_tablet_manager.cc:1397] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 23:57:23.129560 2606 ts_tablet_manager.cc:1397] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:57:23.133255 2605 raft_consensus.cc:348] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.133623 2605 raft_consensus.cc:374] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:23.133687 2605 raft_consensus.cc:729] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770cce1c3e2442aa94bc53bb13ac7caf, State: Initialized, Role: FOLLOWER
I20260928 23:57:23.133548 2604 raft_consensus.cc:348] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.133548 2606 raft_consensus.cc:348] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.133811 2606 raft_consensus.cc:374] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:23.133812 2604 raft_consensus.cc:374] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:23.133862 2606 raft_consensus.cc:729] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864e8b94de4f47b280f2b13e1a947e09, State: Initialized, Role: FOLLOWER
I20260928 23:57:23.133862 2604 raft_consensus.cc:729] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539b4738f4994589b91c6ffca17107ea, State: Initialized, Role: FOLLOWER
I20260928 23:57:23.134224 2605 consensus_queue.cc:260] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.134300 2604 consensus_queue.cc:260] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.134333 2606 consensus_queue.cc:260] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.135423 2604 ts_tablet_manager.cc:1428] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:57:23.135423 2606 ts_tablet_manager.cc:1428] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:57:23.135445 2605 ts_tablet_manager.cc:1428] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:57:23.135821 2453 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
I20260928 23:57:23.135821 2587 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
I20260928 23:57:23.135914 2319 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
W20260928 23:57:23.297418 2588 tablet.cc:2369] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:57:23.310621 2454 tablet.cc:2369] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:57:23.318095 2611 raft_consensus.cc:482] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:23.318332 2611 raft_consensus.cc:504] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.319645 2611 leader_election.cc:288] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:23.324491 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.319921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34823 (local address 127.20.88.2:41657)
0928 23:57:23.320068 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.320089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.320205 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.320599 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.320605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.320630 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:23.320776 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.320785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.322171 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.322175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.323113 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.323123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.323938 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.323942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.324082 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.324123 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.324203 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.324329 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260928 23:57:23.324519 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.320086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:41657 (local address 127.20.88.3:34823)
0928 23:57:23.320217 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.320223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.320270 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:57:23.320348 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.320352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.320410 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.320518 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:23.321007 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.322035 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.323270 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.323783 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.323874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.323941 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.324270 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.324360 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:57:23.324681 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.319921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35115 (local address 127.20.88.2:49459)
0928 23:57:23.320486 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.320503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.320611 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.321006 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.321009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.321030 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:23.321543 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.321557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.322454 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.322457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.323342 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.323353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.324227 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.324231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.324356 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.324381 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.324462 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.324534 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":143,"thread_start_us":143,"threads_started":1}
I20260928 23:57:23.324741 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.320072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:49459 (local address 127.20.88.1:35115)
0928 23:57:23.320216 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.320255 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.320622 (+ 367us) server_negotiation.cc:407] Connection header received
0928 23:57:23.320721 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.320727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.320785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.320879 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:23.321721 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.322327 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.323503 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.324064 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.324197 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.324286 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.324501 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.324609 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:57:23.325302 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29329d64e8446d999d561d199467d1" candidate_uuid: "539b4738f4994589b91c6ffca17107ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" is_pre_election: true
I20260928 23:57:23.325305 2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29329d64e8446d999d561d199467d1" candidate_uuid: "539b4738f4994589b91c6ffca17107ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864e8b94de4f47b280f2b13e1a947e09" is_pre_election: true
I20260928 23:57:23.325675 2273 raft_consensus.cc:2454] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539b4738f4994589b91c6ffca17107ea in term 0.
I20260928 23:57:23.325675 2541 raft_consensus.cc:2454] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539b4738f4994589b91c6ffca17107ea in term 0.
I20260928 23:57:23.326264 2341 leader_election.cc:302] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 539b4738f4994589b91c6ffca17107ea, 770cce1c3e2442aa94bc53bb13ac7caf, 864e8b94de4f47b280f2b13e1a947e09; no voters:
I20260928 23:57:23.326587 2611 raft_consensus.cc:2781] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:57:23.326727 2611 raft_consensus.cc:482] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:57:23.326812 2611 raft_consensus.cc:3037] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:23.328150 2611 raft_consensus.cc:504] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.328635 2611 leader_election.cc:288] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [CANDIDATE]: Term 1 election: Requested vote from peers 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:23.329201 2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29329d64e8446d999d561d199467d1" candidate_uuid: "539b4738f4994589b91c6ffca17107ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864e8b94de4f47b280f2b13e1a947e09"
I20260928 23:57:23.329201 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d29329d64e8446d999d561d199467d1" candidate_uuid: "539b4738f4994589b91c6ffca17107ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
I20260928 23:57:23.329384 2541 raft_consensus.cc:3037] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:23.329375 2273 raft_consensus.cc:3037] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:23.330790 2273 raft_consensus.cc:2454] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539b4738f4994589b91c6ffca17107ea in term 1.
I20260928 23:57:23.330790 2541 raft_consensus.cc:2454] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539b4738f4994589b91c6ffca17107ea in term 1.
I20260928 23:57:23.331215 2343 leader_election.cc:302] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 539b4738f4994589b91c6ffca17107ea, 770cce1c3e2442aa94bc53bb13ac7caf, 864e8b94de4f47b280f2b13e1a947e09; no voters:
I20260928 23:57:23.331427 2611 raft_consensus.cc:2781] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:23.331806 2611 raft_consensus.cc:686] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 1 LEADER]: Becoming Leader. State: Replica: 539b4738f4994589b91c6ffca17107ea, State: Running, Role: LEADER
I20260928 23:57:23.332218 2611 consensus_queue.cc:237] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
W20260928 23:57:23.334404 2320 tablet.cc:2369] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:57:23.335954 2128 catalog_manager.cc:5495] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 539b4738f4994589b91c6ffca17107ea (127.20.88.2). New cstate: current_term: 1 leader_uuid: "539b4738f4994589b91c6ffca17107ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } health_report { overall_health: UNKNOWN } } }
I20260928 23:57:23.348438 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.344851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.20.88.2:41875)
0928 23:57:23.345019 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.345022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.345037 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:57:23.345153 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.345155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.345226 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.345318 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:57:23.345610 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.346107 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.346939 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.347145 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.347238 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.347491 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:57:23.347923 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:57:23.348014 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.348128 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.348224 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 23:57:23.354583 2541 raft_consensus.cc:1261] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Refusing update from remote peer 539b4738f4994589b91c6ffca17107ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:23.354606 2273 raft_consensus.cc:1261] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Refusing update from remote peer 539b4738f4994589b91c6ffca17107ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:23.355366 2611 consensus_queue.cc:1035] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:23.355615 2614 consensus_queue.cc:1035] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:23.363946 2619 mvcc.cc:205] Tried to move back new op lower bound from 7334460798373056512 to 7334460798291972096. Current Snapshot: MvccSnapshot[applied={T|T < 7334460798373056512}]
I20260928 23:57:23.364176 2618 mvcc.cc:205] Tried to move back new op lower bound from 7334460798373056512 to 7334460798291972096. Current Snapshot: MvccSnapshot[applied={T|T < 7334460798373056512}]
I20260928 23:57:23.372342 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:23.372551 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:23.375541 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Sending DeleteTablet for 3 replicas of tablet 4d29329d64e8446d999d561d199467d1
I20260928 23:57:23.376827 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29329d64e8446d999d561d199467d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:23 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55118
I20260928 23:57:23.377060 2521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29329d64e8446d999d561d199467d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:23 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55148
I20260928 23:57:23.377207 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2190
I20260928 23:57:23.378132 2627 tablet_replica.cc:330] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: stopping tablet replica
I20260928 23:57:23.378549 2626 tablet_replica.cc:330] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: stopping tablet replica
I20260928 23:57:23.378552 2627 raft_consensus.cc:2229] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:23.378928 2627 raft_consensus.cc:2258] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:23.378945 2626 raft_consensus.cc:2229] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:57:23.379487 2626 raft_consensus.cc:2258] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:23.380831 2627 ts_tablet_manager.cc:1905] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:23.381354 2626 ts_tablet_manager.cc:1905] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:23.384274 2626 ts_tablet_manager.cc:1918] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:23.384481 2626 log.cc:1196] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/4d29329d64e8446d999d561d199467d1
I20260928 23:57:23.384531 2627 ts_tablet_manager.cc:1918] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:23.384646 2627 log.cc:1196] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/4d29329d64e8446d999d561d199467d1
I20260928 23:57:23.384987 2626 ts_tablet_manager.cc:1939] T 4d29329d64e8446d999d561d199467d1 P 539b4738f4994589b91c6ffca17107ea: Deleting consensus metadata
I20260928 23:57:23.384992 2627 ts_tablet_manager.cc:1939] T 4d29329d64e8446d999d561d199467d1 P 864e8b94de4f47b280f2b13e1a947e09: Deleting consensus metadata
I20260928 23:57:23.386505 2113 catalog_manager.cc:4841] TS 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): tablet 4d29329d64e8446d999d561d199467d1 (table test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]) successfully deleted
I20260928 23:57:23.386490 2112 catalog_manager.cc:4841] TS 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): tablet 4d29329d64e8446d999d561d199467d1 (table test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]) successfully deleted
W20260928 23:57:23.386781 2115 connection.cc:535] client connection to 127.20.88.1:35115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 23:57:23.386925 2115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:57:23.386940 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2324
I20260928 23:57:23.388811 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.388318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35115 (local address 127.0.0.1:52846)
0928 23:57:23.388644 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.388731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 23:57:23.389029 2115 catalog_manager.cc:4584] TS 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d29329d64e8446d999d561d199467d1: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35115: connect: Connection refused (error 111)
I20260928 23:57:23.393098 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2458
I20260928 23:57:23.398859 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2096
I20260928 23:57:23.405797 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--webserver_interface=127.20.88.62
--webserver_port=35067
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:23.528726 2629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:23.529080 2629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:23.529165 2629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:23.533453 2629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:23.533560 2629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:23.533598 2629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:23.533658 2629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:23.533696 2629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:23.539490 2629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:23.540803 2629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:23.541930 2629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:23.556089 2638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:23.556119 2635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:23.556069 2636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:23.556422 2629 server_base.cc:1034] running on GCE node
I20260928 23:57:23.557193 2629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:23.558461 2629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:23.559664 2629 hybrid_clock.cc:648] HybridClock initialized: now 1790639843559645 us; error 33 us; skew 500 ppm
Sep 28 23:57:23 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639843, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:23.562822 2629 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:23.563822 2629 webserver.cc:460] Webserver started at http://127.20.88.62:35067/ using document root <none> and password file <none>
I20260928 23:57:23.564546 2629 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:23.564667 2629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:23.568811 2629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:23.571273 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:23.572813 2629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:23.572988 2629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "07aa4c60080141ef9e9264c5d70816d7"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "88814a16cdd09cd9a391e3f0598487cc"
server_key_iv: "4da40041b4877f9f5f938abab0459edf"
server_key_version: "encryptionkey@0"
I20260928 23:57:23.573664 2629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:23.609081 2629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:23.612732 2629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:23.612913 2629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:23.620791 2629 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:46439
I20260928 23:57:23.621100 2697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:46439 every 8 connection(s)
I20260928 23:57:23.622473 2629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:23.623917 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2629
I20260928 23:57:23.624459 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:35115
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=42255
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:23.628124 2698 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:57:23.631156 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap starting.
I20260928 23:57:23.655496 2698 log.cc:824] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:23.666985 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:57:23.667464 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap complete.
I20260928 23:57:23.673383 2698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:23.673987 2698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Initialized, Role: FOLLOWER
I20260928 23:57:23.674567 2698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:23.674726 2698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:23.674803 2698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:23.674901 2698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:57:23.677721 2698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:23.678144 2698 leader_election.cc:302] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07aa4c60080141ef9e9264c5d70816d7; no voters:
I20260928 23:57:23.678531 2698 leader_election.cc:288] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:57:23.678706 2702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:57:23.679008 2702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 LEADER]: Becoming Leader. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Running, Role: LEADER
I20260928 23:57:23.679345 2702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:23.679875 2698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:23.681061 2703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:23.681195 2703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:23.681077 2704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07aa4c60080141ef9e9264c5d70816d7. Latest consensus state: current_term: 2 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:23.681602 2704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:23.683717 2711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:23.688047 2711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:23.688702 2711 catalog_manager.cc:1259] Loaded cluster ID: bebf4e8be7384ebf9d789c7710bb457c
I20260928 23:57:23.688786 2711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:23.690202 2711 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 23:57:23.693037 2711 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 23:57:23.698378 2711 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:23.699261 2711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Loaded TSK: 0
I20260928 23:57:23.700508 2711 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:57:23.775614 2700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:23.776036 2700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:23.776129 2700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:23.781306 2700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:23.781411 2700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:23.781519 2700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:23.787505 2700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:35115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=42255
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:23.788784 2700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:23.789979 2700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:23.800643 2726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:23.800819 2728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:23.801102 2700 server_base.cc:1034] running on GCE node
W20260928 23:57:23.800643 2725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:23.801749 2700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:23.802649 2700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:23.803884 2700 hybrid_clock.cc:648] HybridClock initialized: now 1790639843803825 us; error 67 us; skew 500 ppm
Sep 28 23:57:23 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639843, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:23.807725 2700 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:23.809055 2700 webserver.cc:460] Webserver started at http://127.20.88.1:42255/ using document root <none> and password file <none>
I20260928 23:57:23.809860 2700 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:23.809989 2700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:23.815828 2700 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:57:23.818903 2735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:23.820720 2700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:57:23.820879 2700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "a4d14c6334cd1b3b6fd44a11e68cf1c5"
server_key_iv: "163a2000bd572106234580c0dcadfd18"
server_key_version: "encryptionkey@0"
I20260928 23:57:23.821698 2700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:23.837867 2700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:23.842168 2700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:23.842410 2700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:23.843251 2700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:23.844815 2742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:57:23.849318 2700 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:57:23.849411 2700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 23:57:23.849481 2700 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:57:23.851776 2700 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:57:23.851847 2700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:57:23.852285 2742 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap starting.
I20260928 23:57:23.862746 2700 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35115
I20260928 23:57:23.863390 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35115 every 8 connection(s)
I20260928 23:57:23.863978 2700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:23.873482 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2700
I20260928 23:57:23.873941 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:41875
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=44119
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:23.873904 2742 log.cc:824] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:57:23 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639843, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:23.883198 2742 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:57:23.883682 2742 tablet_bootstrap.cc:492] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap complete.
I20260928 23:57:23.884404 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.866390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:57773)
0928 23:57:23.867065 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.867152 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.868369 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.869682 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.869694 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.870331 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:23.871066 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.871092 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.872620 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.872624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.874003 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.874016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.874954 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.874958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.875156 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.875844 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:23.875861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:23.877675 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:23.880566 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.880572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.880584 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.880969 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.881439 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.881443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.881445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.881611 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.882541 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:23.882548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:23.882834 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.883031 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.883378 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":194,"threads_started":1}
I20260928 23:57:23.884567 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.866376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57773 (local address 127.20.88.62:46439)
0928 23:57:23.866810 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.866830 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.867310 (+ 480us) server_negotiation.cc:407] Connection header received
0928 23:57:23.868878 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.868891 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.869224 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.869510 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:23.871333 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.872471 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.874169 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.874785 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.874990 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.877930 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:23.877937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:23.877958 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:23.878002 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:23.880400 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.881118 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.881128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.881137 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.881322 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.881779 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.881785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.881789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.882405 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:23.882564 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.883201 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.883558 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260928 23:57:23.885351 2742 ts_tablet_manager.cc:1397] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.015s
I20260928 23:57:23.887154 2850 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
I20260928 23:57:23.887385 2850 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:23.887948 2850 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:23.888445 2742 raft_consensus.cc:348] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.888963 2742 raft_consensus.cc:729] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770cce1c3e2442aa94bc53bb13ac7caf, State: Initialized, Role: FOLLOWER
I20260928 23:57:23.889465 2742 consensus_queue.cc:260] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } } peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } }
I20260928 23:57:23.890609 2742 ts_tablet_manager.cc:1428] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:57:23.890847 2658 ts_manager.cc:194] Registered new tserver with Master: 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115)
I20260928 23:57:23.894852 2658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:57773
I20260928 23:57:23.898232 2850 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
Sep 28 23:57:23 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639843, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:57:23.907881 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.894230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35115 (local address 127.0.0.1:52848)
0928 23:57:23.894682 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:23.894737 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:23.894893 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:23.895674 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:23.895680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:23.896192 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:23.896391 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.896403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.898513 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.898518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.900019 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:23.900032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.901174 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.901178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:23.901276 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.901983 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:23.901996 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:23.903648 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:23.905951 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.905960 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.905964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.906207 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.906538 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:23.906540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:23.906541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:23.906618 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:23.907202 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:23.907208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:23.907343 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 23:57:23.907532 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 23:57:23.907730 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260928 23:57:23.907967 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:23.894349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.20.88.1:35115)
0928 23:57:23.895151 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:23.895170 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:23.895195 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:57:23.895284 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:23.895290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:23.895501 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:23.895641 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:23.896549 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.898115 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.900263 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:23.900916 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:23.901022 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:23.903818 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:23.903831 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:23.903845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:23.903874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:23.905798 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.906337 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.906344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.906346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.906444 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:23.906733 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:23.906736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:23.906737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:23.907064 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:23.907197 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:23.907602 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 23:57:23.907799 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":104,"threads_started":1}
I20260928 23:57:23.909274 2784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d29329d64e8446d999d561d199467d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:23 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:52848
I20260928 23:57:23.910308 2859 tablet_replica.cc:330] stopping tablet replica
I20260928 23:57:23.910647 2859 raft_consensus.cc:2229] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:23.910923 2859 raft_consensus.cc:2258] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:23.911976 2859 ts_tablet_manager.cc:1905] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:23.915930 2859 ts_tablet_manager.cc:1918] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:23.916093 2859 log.cc:1196] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/4d29329d64e8446d999d561d199467d1
I20260928 23:57:23.916457 2859 ts_tablet_manager.cc:1939] T 4d29329d64e8446d999d561d199467d1 P 770cce1c3e2442aa94bc53bb13ac7caf: Deleting consensus metadata
I20260928 23:57:23.918417 2649 catalog_manager.cc:4841] TS 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115): tablet 4d29329d64e8446d999d561d199467d1 (table test-table [id=6ca6e26cd7a342e59db17bee27ebe9e0]) successfully deleted
W20260928 23:57:24.003494 2855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:24.003791 2855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:24.003837 2855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:24.008044 2855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:24.008124 2855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:24.008193 2855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:24.013952 2855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:41875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=44119
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:24.015021 2855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:24.016009 2855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:24.025790 2865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.025782 2864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.025913 2855 server_base.cc:1034] running on GCE node
W20260928 23:57:24.025789 2867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.026532 2855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:24.027143 2855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:24.028326 2855 hybrid_clock.cc:648] HybridClock initialized: now 1790639844028304 us; error 31 us; skew 500 ppm
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:24.031158 2855 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:24.032109 2855 webserver.cc:460] Webserver started at http://127.20.88.2:44119/ using document root <none> and password file <none>
I20260928 23:57:24.032806 2855 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:24.032931 2855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:24.037141 2855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:57:24.039399 2874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.040882 2855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:24.041000 2855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "539b4738f4994589b91c6ffca17107ea"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "4c87162e502def35a3a44b852ac7fad6"
server_key_iv: "33ffee6d625563e0545114a253609954"
server_key_version: "encryptionkey@0"
I20260928 23:57:24.041638 2855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:24.061987 2855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:24.065452 2855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:24.065654 2855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:24.066344 2855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:24.067237 2855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:24.067288 2855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.067329 2855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:24.067345 2855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.077348 2855 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:41875
I20260928 23:57:24.077381 2987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:41875 every 8 connection(s)
I20260928 23:57:24.078469 2855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:24.084498 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2855
I20260928 23:57:24.084926 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:34823
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=35517
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:24.093168 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.080428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:43903 (local address 127.20.88.62:46439)
0928 23:57:24.080603 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.080607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.081582 (+ 975us) server_negotiation.cc:407] Connection header received
0928 23:57:24.083092 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.083096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.083172 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.083281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:24.084567 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.085247 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.086414 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.086669 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.086782 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.089311 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:24.089315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:24.089318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:24.089354 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:24.090934 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.091502 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.091506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.091508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.091593 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.091943 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.091947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.091949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.092146 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:24.092293 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.092836 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.092976 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:57:24.093981 2990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.080901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:43903)
0928 23:57:24.081439 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.081483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.082872 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.083467 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.083474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.083850 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:24.084414 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.084431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.085383 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.085387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.086276 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.086285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.086826 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.086830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.087008 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.087638 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:24.087661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:24.089129 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:24.091089 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:24.091095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:24.091112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:24.091372 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:24.091691 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:24.091695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:24.091697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:24.091842 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:24.092315 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:24.092321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:24.092583 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.092779 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.093114 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260928 23:57:24.095295 2988 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
I20260928 23:57:24.095597 2988 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:24.096184 2988 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:24.097388 2658 ts_manager.cc:194] Registered new tserver with Master: 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875)
I20260928 23:57:24.098156 2658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:43903
W20260928 23:57:24.208063 2991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:24.208465 2991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:24.208519 2991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:24.212837 2991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:24.212914 2991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:24.212980 2991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:24.218669 2991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:34823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=35517
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:24.219843 2991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:24.220970 2991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:24.231611 2997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.231665 2996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.231649 2999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.231900 2991 server_base.cc:1034] running on GCE node
I20260928 23:57:24.232483 2991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:24.233135 2991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:24.234338 2991 hybrid_clock.cc:648] HybridClock initialized: now 1790639844234312 us; error 37 us; skew 500 ppm
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:24.237426 2991 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:24.238457 2991 webserver.cc:460] Webserver started at http://127.20.88.3:35517/ using document root <none> and password file <none>
I20260928 23:57:24.239073 2991 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:24.239192 2991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:24.243742 2991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:57:24.246438 3006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.247855 2991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:57:24.248041 2991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "864e8b94de4f47b280f2b13e1a947e09"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "8801ac5e9c8281ef2dcd4f8e18874cec"
server_key_iv: "3411e2126b56daea44954ec422445c35"
server_key_version: "encryptionkey@0"
I20260928 23:57:24.248736 2991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:24.285903 2991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:24.289603 2991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:24.289846 2991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:24.290560 2991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:24.291507 2991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:24.291582 2991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.291666 2991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:24.291715 2991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.301949 2991 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:34823
I20260928 23:57:24.301970 3119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:34823 every 8 connection(s)
I20260928 23:57:24.303098 2991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:24.313010 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 2991
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:24.318786 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.305304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:43721 (local address 127.20.88.62:46439)
0928 23:57:24.305458 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.305462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.306350 (+ 888us) server_negotiation.cc:407] Connection header received
0928 23:57:24.307845 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.307849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.307915 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.308022 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:24.309358 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.309893 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.311129 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.311369 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.311491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.314977 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:24.314981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:24.314983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:24.315007 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:24.316574 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.317143 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.317147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.317149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.317234 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.317608 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.317612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.317614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.317813 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:24.317925 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.318467 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.318610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260928 23:57:24.319684 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.305678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:43721)
0928 23:57:24.306180 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.306226 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.307591 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.308223 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.308259 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.308664 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:24.309182 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.309196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.310036 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.310047 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.310974 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.310984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.311543 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.311547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.311740 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.312568 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:24.312632 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:24.314778 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:24.316701 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:24.316708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:24.316728 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:24.317032 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:24.317336 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:24.317339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:24.317341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:24.317505 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:24.317923 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:24.317933 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:24.318223 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.318427 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.318806 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260928 23:57:24.321274 3120 heartbeater.cc:347] Connected to a master server at 127.20.88.62:46439
I20260928 23:57:24.321591 3120 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:24.322129 3120 heartbeater.cc:511] Master 127.20.88.62:46439 requested a full tablet report, sending...
I20260928 23:57:24.323295 2658 ts_manager.cc:194] Registered new tserver with Master: 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:24.324003 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.313952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.20.88.62:46439)
0928 23:57:24.314335 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.314342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.314454 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:57:24.314658 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.314662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.314727 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.314843 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:24.315740 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.316436 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.317460 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.317754 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.317942 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.319830 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:24.319836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:24.319839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:24.319879 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:24.321852 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.322397 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.322403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.322405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.322516 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:24.322865 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:24.322869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:24.322871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:24.323163 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:24.323398 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.323652 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.323799 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260928 23:57:24.324158 2658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:43721
I20260928 23:57:24.326242 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:57:24.330825 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.326950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.20.88.62:46439)
0928 23:57:24.327159 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.327162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.327376 (+ 214us) server_negotiation.cc:407] Connection header received
0928 23:57:24.327493 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.327496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.327589 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.327685 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:57:24.327908 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.328586 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.329467 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.329716 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.329798 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.330078 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:57:24.330485 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:57:24.330576 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.330630 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.330716 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 23:57:24.331496 2662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:57:24.335178 2662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:57:24.344276 2784 tablet_service.cc:1467] Processing CreateTablet for tablet 7e03289cc820469a9a3b30edf11828a4 (DEFAULT_TABLE table=test-table [id=1f0253a1851a4668a3615f8c0fa5cf37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:24.345072 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e03289cc820469a9a3b30edf11828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:24.348292 2742 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap starting.
I20260928 23:57:24.348743 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.342947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41875 (local address 127.0.0.1:55142)
0928 23:57:24.343094 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.343110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.343200 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.344153 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.344156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.344178 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:24.344339 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.344352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.346001 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.346014 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.347043 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.347054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.348261 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.348265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.348388 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.348450 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.348502 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.348592 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 23:57:24.349272 2742 tablet_bootstrap.cc:654] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:24.349399 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.343144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.20.88.2:41875)
0928 23:57:24.343549 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.343563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.343588 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:57:24.343676 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.343680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.343953 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.344516 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:24.344696 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.345869 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.347188 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.348063 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.348957 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.349109 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.349164 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.349243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":139,"threads_started":1}
I20260928 23:57:24.350807 2742 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: No bootstrap required, opened a new log
I20260928 23:57:24.350908 2742 ts_tablet_manager.cc:1397] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:57:24.351316 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.343642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34823 (local address 127.0.0.1:55158)
0928 23:57:24.343940 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.343953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.344041 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.345141 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.345144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.345163 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:24.345272 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.345279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.346908 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.346912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.347800 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.347812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.350832 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.350836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.350940 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.350966 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.351053 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.351166 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260928 23:57:24.351361 2742 raft_consensus.cc:348] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.351485 2742 raft_consensus.cc:374] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:24.351521 2742 raft_consensus.cc:729] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770cce1c3e2442aa94bc53bb13ac7caf, State: Initialized, Role: FOLLOWER
I20260928 23:57:24.351661 2742 consensus_queue.cc:260] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.351732 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.343841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.20.88.3:34823)
0928 23:57:24.344487 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.344500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.344523 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:57:24.344598 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.344603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.344814 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.345431 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:24.345642 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.346762 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.347955 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.350684 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.350817 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.350967 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.351136 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.351244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":177,"threads_started":1}
I20260928 23:57:24.352749 2742 ts_tablet_manager.cc:1428] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:57:24.352772 2922 tablet_service.cc:1467] Processing CreateTablet for tablet 7e03289cc820469a9a3b30edf11828a4 (DEFAULT_TABLE table=test-table [id=1f0253a1851a4668a3615f8c0fa5cf37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:24.354066 2922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e03289cc820469a9a3b30edf11828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:24.355145 3054 tablet_service.cc:1467] Processing CreateTablet for tablet 7e03289cc820469a9a3b30edf11828a4 (DEFAULT_TABLE table=test-table [id=1f0253a1851a4668a3615f8c0fa5cf37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:24.356324 3054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e03289cc820469a9a3b30edf11828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:24.362695 3133 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Bootstrap starting.
I20260928 23:57:24.364276 3134 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Bootstrap starting.
I20260928 23:57:24.365513 3133 tablet_bootstrap.cc:654] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Neither blocks nor log segments found. Creating new log.
W20260928 23:57:24.365690 2851 tablet.cc:2369] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:57:24.366441 3134 tablet_bootstrap.cc:654] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:24.366511 3133 log.cc:824] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:24.367758 3134 log.cc:824] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:24.368803 3133 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: No bootstrap required, opened a new log
I20260928 23:57:24.369031 3133 ts_tablet_manager.cc:1397] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:57:24.369642 3134 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: No bootstrap required, opened a new log
I20260928 23:57:24.369840 3134 ts_tablet_manager.cc:1397] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:57:24.372138 3133 raft_consensus.cc:348] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.372524 3133 raft_consensus.cc:374] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:24.372612 3133 raft_consensus.cc:729] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539b4738f4994589b91c6ffca17107ea, State: Initialized, Role: FOLLOWER
I20260928 23:57:24.372799 3134 raft_consensus.cc:348] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.373009 3134 raft_consensus.cc:374] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:24.373088 3134 raft_consensus.cc:729] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864e8b94de4f47b280f2b13e1a947e09, State: Initialized, Role: FOLLOWER
I20260928 23:57:24.373162 3133 consensus_queue.cc:260] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.373514 3134 consensus_queue.cc:260] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.374185 2988 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
I20260928 23:57:24.374518 3134 ts_tablet_manager.cc:1428] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:57:24.375277 3120 heartbeater.cc:503] Master 127.20.88.62:46439 was elected leader, sending a full tablet report...
I20260928 23:57:24.376991 3133 ts_tablet_manager.cc:1428] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:57:24.467662 2856 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:24.467900 2856 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.469250 2856 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:24.474802 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.469634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:36481 (local address 127.20.88.2:41875)
0928 23:57:24.469771 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.469776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.469993 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:57:24.470142 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.470146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.470207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.470302 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:24.470750 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.471540 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.473403 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.474027 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.474145 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.474225 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.474584 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.474664 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:57:24.474802 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.469966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:41225 (local address 127.20.88.3:34823)
0928 23:57:24.470117 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.470121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.470134 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:57:24.470181 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.470184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.470247 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.470323 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:24.470750 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.471540 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.473403 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.474027 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.474149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.474217 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.474584 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.474689 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:57:24.474860 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.469583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34823 (local address 127.20.88.1:41225)
0928 23:57:24.469878 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.469891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.470027 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.470377 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.470380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.470420 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:24.470606 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.470612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.471676 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.471680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.473250 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.473264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.474170 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.474174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.474350 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.474443 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.474525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.474645 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":311,"spinlock_wait_cycles":56960,"thread_start_us":63,"threads_started":1}
I20260928 23:57:24.474864 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.469482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41875 (local address 127.20.88.1:36481)
0928 23:57:24.469874 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.469890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:24.469999 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:24.470377 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:24.470380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:24.470405 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:24.470598 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.470608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.471676 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.471680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.473250 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:24.473264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.474170 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.474174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:24.474355 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.474448 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:57:24.474525 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:57:24.474645 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":225,"spinlock_wait_cycles":2304,"thread_start_us":155,"threads_started":1}
I20260928 23:57:24.475749 3074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e03289cc820469a9a3b30edf11828a4" candidate_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864e8b94de4f47b280f2b13e1a947e09" is_pre_election: true
I20260928 23:57:24.475752 2942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e03289cc820469a9a3b30edf11828a4" candidate_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539b4738f4994589b91c6ffca17107ea" is_pre_election: true
I20260928 23:57:24.476070 2942 raft_consensus.cc:2454] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770cce1c3e2442aa94bc53bb13ac7caf in term 0.
I20260928 23:57:24.476083 3074 raft_consensus.cc:2454] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770cce1c3e2442aa94bc53bb13ac7caf in term 0.
I20260928 23:57:24.476634 2736 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539b4738f4994589b91c6ffca17107ea, 770cce1c3e2442aa94bc53bb13ac7caf; no voters:
I20260928 23:57:24.476946 2856 raft_consensus.cc:2781] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:57:24.477041 2856 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:57:24.477084 2856 raft_consensus.cc:3037] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:24.478264 2856 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.478658 2856 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 1 election: Requested vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
I20260928 23:57:24.479071 2942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e03289cc820469a9a3b30edf11828a4" candidate_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539b4738f4994589b91c6ffca17107ea"
I20260928 23:57:24.479171 3074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e03289cc820469a9a3b30edf11828a4" candidate_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864e8b94de4f47b280f2b13e1a947e09"
I20260928 23:57:24.479272 2942 raft_consensus.cc:3037] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:24.479342 3074 raft_consensus.cc:3037] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:24.480572 2942 raft_consensus.cc:2454] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770cce1c3e2442aa94bc53bb13ac7caf in term 1.
I20260928 23:57:24.480671 3074 raft_consensus.cc:2454] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770cce1c3e2442aa94bc53bb13ac7caf in term 1.
I20260928 23:57:24.480893 2736 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539b4738f4994589b91c6ffca17107ea, 770cce1c3e2442aa94bc53bb13ac7caf; no voters:
I20260928 23:57:24.481069 2856 raft_consensus.cc:2781] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:24.481371 2856 raft_consensus.cc:686] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 LEADER]: Becoming Leader. State: Replica: 770cce1c3e2442aa94bc53bb13ac7caf, State: Running, Role: LEADER
I20260928 23:57:24.481673 2856 consensus_queue.cc:237] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:24.484959 2662 catalog_manager.cc:5495] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf reported cstate change: term changed from 0 to 1, leader changed from <none> to 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1). New cstate: current_term: 1 leader_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } health_report { overall_health: HEALTHY } } }
I20260928 23:57:24.509554 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.505572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.20.88.1:35115)
0928 23:57:24.505931 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:24.505939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:24.505951 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:57:24.506030 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:24.506032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:24.506089 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:24.506243 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:57:24.506505 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.507042 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.507929 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:24.508162 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:24.508301 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:24.508599 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:57:24.509093 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:57:24.509229 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:24.509298 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:57:24.509364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
I20260928 23:57:24.514163 3074 raft_consensus.cc:1261] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Refusing update from remote peer 770cce1c3e2442aa94bc53bb13ac7caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:24.514163 2942 raft_consensus.cc:1261] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Refusing update from remote peer 770cce1c3e2442aa94bc53bb13ac7caf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:24.515071 3141 consensus_queue.cc:1035] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:24.515304 2856 consensus_queue.cc:1035] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:24.526016 3146 mvcc.cc:205] Tried to move back new op lower bound from 7334460803122278400 to 7334460802999603200. Current Snapshot: MvccSnapshot[applied={T|T < 7334460803122278400}]
I20260928 23:57:24.526624 3147 mvcc.cc:205] Tried to move back new op lower bound from 7334460803122278400 to 7334460802999603200. Current Snapshot: MvccSnapshot[applied={T|T < 7334460803122278400}]
I20260928 23:57:24.533393 2662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:24.533596 2662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:24.538734 2662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Sending DeleteTablet for 3 replicas of tablet 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:24.539577 2784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e03289cc820469a9a3b30edf11828a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:52848
I20260928 23:57:24.539603 2922 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e03289cc820469a9a3b30edf11828a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55142
I20260928 23:57:24.539688 3054 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e03289cc820469a9a3b30edf11828a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:24 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55158
I20260928 23:57:24.540335 3155 tablet_replica.cc:330] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: stopping tablet replica
I20260928 23:57:24.540416 3154 tablet_replica.cc:330] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: stopping tablet replica
I20260928 23:57:24.540483 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2700
I20260928 23:57:24.540560 3154 raft_consensus.cc:2229] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:57:24.540925 3155 raft_consensus.cc:2229] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:24.541519 3155 raft_consensus.cc:2258] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:24.541966 3156 tablet_replica.cc:330] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: stopping tablet replica
I20260928 23:57:24.542330 3156 raft_consensus.cc:2229] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:24.542670 3156 raft_consensus.cc:2258] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:24.543677 3155 ts_tablet_manager.cc:1905] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:24.544023 3156 ts_tablet_manager.cc:1905] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:24.546957 3156 ts_tablet_manager.cc:1918] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:24.547111 3156 log.cc:1196] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:24.547626 3156 ts_tablet_manager.cc:1939] T 7e03289cc820469a9a3b30edf11828a4 P 864e8b94de4f47b280f2b13e1a947e09: Deleting consensus metadata
I20260928 23:57:24.547667 3155 ts_tablet_manager.cc:1918] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:24.547806 3155 log.cc:1196] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:24.548287 3155 ts_tablet_manager.cc:1939] T 7e03289cc820469a9a3b30edf11828a4 P 539b4738f4994589b91c6ffca17107ea: Deleting consensus metadata
I20260928 23:57:24.549494 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2855
W20260928 23:57:24.549479 2649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:35115 (error 108)
I20260928 23:57:24.550469 2647 catalog_manager.cc:4841] TS 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): tablet 7e03289cc820469a9a3b30edf11828a4 (table test-table [id=1f0253a1851a4668a3615f8c0fa5cf37]) successfully deleted
I20260928 23:57:24.551153 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.550772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:35115 (local address 127.0.0.1:52860)
0928 23:57:24.550937 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.551028 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 23:57:24.551322 2649 catalog_manager.cc:4584] TS 770cce1c3e2442aa94bc53bb13ac7caf (127.20.88.1:35115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e03289cc820469a9a3b30edf11828a4: Network error: Client connection negotiation failed: client connection to 127.20.88.1:35115: connect: Connection refused (error 111)
I20260928 23:57:24.555949 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2991
I20260928 23:57:24.557426 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:24.557132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41875 (local address 127.0.0.1:55146)
0928 23:57:24.557274 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:24.557333 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:41875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 23:57:24.561982 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 2629
I20260928 23:57:24.568764 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--webserver_interface=127.20.88.62
--webserver_port=35067
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:24.686407 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:24.686709 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:24.686754 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:24.690995 3158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:24.691087 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:24.691108 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:24.691125 3158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:24.691141 3158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:24.697356 3158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=35067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:24.698537 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:24.699596 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:24.709192 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.709194 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.709211 3158 server_base.cc:1034] running on GCE node
W20260928 23:57:24.709254 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.710155 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:24.711467 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:24.712647 3158 hybrid_clock.cc:648] HybridClock initialized: now 1790639844712627 us; error 40 us; skew 500 ppm
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:24.715695 3158 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:24.716794 3158 webserver.cc:460] Webserver started at http://127.20.88.62:35067/ using document root <none> and password file <none>
I20260928 23:57:24.717407 3158 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:24.717473 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:24.721637 3158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:57:24.724069 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.725498 3158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:24.725670 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "07aa4c60080141ef9e9264c5d70816d7"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "88814a16cdd09cd9a391e3f0598487cc"
server_key_iv: "4da40041b4877f9f5f938abab0459edf"
server_key_version: "encryptionkey@0"
I20260928 23:57:24.726290 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:24.744663 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:24.747881 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:24.748139 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:24.756137 3158 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:46439
I20260928 23:57:24.756433 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:46439 every 8 connection(s)
I20260928 23:57:24.757714 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:24.762491 3226 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:57:24.764389 3226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap starting.
I20260928 23:57:24.766322 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3158
I20260928 23:57:24.766776 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:35115
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=42255
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:24.784487 3226 log.cc:824] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:24.798163 3226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:57:24.798672 3226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7: Bootstrap complete.
I20260928 23:57:24.803193 3226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:24.803651 3226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Initialized, Role: FOLLOWER
I20260928 23:57:24.804224 3226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:24.804430 3226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:24.804513 3226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:24.804629 3226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:57:24.806949 3226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:24.807356 3226 leader_election.cc:302] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07aa4c60080141ef9e9264c5d70816d7; no voters:
I20260928 23:57:24.807755 3226 leader_election.cc:288] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:57:24.807922 3230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:57:24.808151 3230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [term 3 LEADER]: Becoming Leader. State: Replica: 07aa4c60080141ef9e9264c5d70816d7, State: Running, Role: LEADER
I20260928 23:57:24.808521 3230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } }
I20260928 23:57:24.809249 3226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:24.809612 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07aa4c60080141ef9e9264c5d70816d7. Latest consensus state: current_term: 3 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:24.809760 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:24.809955 3231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "07aa4c60080141ef9e9264c5d70816d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07aa4c60080141ef9e9264c5d70816d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46439 } } }
I20260928 23:57:24.810061 3231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07aa4c60080141ef9e9264c5d70816d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:24.810165 3234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:24.816109 3234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:24.816989 3234 catalog_manager.cc:1259] Loaded cluster ID: bebf4e8be7384ebf9d789c7710bb457c
I20260928 23:57:24.817090 3234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 23:57:24.818017 3234 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f04774469ad google::LogMessage::Fail() at ??:0
@ 0x7f047744988a google::LogMessage::SendToLog() at ??:0
@ 0x7f04774464de google::LogMessage::Flush() at ??:0
@ 0x7f0477447a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0481d85fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0481d86d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0481d83579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0481dc7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f04796983d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0479698c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f047969a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0479689512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f047accd6db start_thread at ??:0
@ 0x7f0475f4461f clone at ??:0
W20260928 23:57:24.922043 3228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:24.922483 3228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:24.922559 3228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:24.929092 3228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:24.929208 3228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:24.929303 3228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:24.938172 3228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:35115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=42255
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:24.939756 3228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:24.941241 3228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:24.953631 3268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.953663 3267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:24.953631 3270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:24.954228 3228 server_base.cc:1034] running on GCE node
I20260928 23:57:24.954762 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:24.955495 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:24.956707 3228 hybrid_clock.cc:648] HybridClock initialized: now 1790639844956673 us; error 51 us; skew 500 ppm
Sep 28 23:57:24 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:24.959923 3228 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:24.961087 3228 webserver.cc:460] Webserver started at http://127.20.88.1:42255/ using document root <none> and password file <none>
I20260928 23:57:24.961719 3228 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:24.961792 3228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:24.966382 3228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:57:24.969076 3277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:24.970681 3228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:24.970866 3228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "770cce1c3e2442aa94bc53bb13ac7caf"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "a4d14c6334cd1b3b6fd44a11e68cf1c5"
server_key_iv: "163a2000bd572106234580c0dcadfd18"
server_key_version: "encryptionkey@0"
I20260928 23:57:24.971495 3228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:24.991102 3228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:24.994874 3228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:24.995108 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:24.995782 3228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:24.997148 3284 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:57:25.001748 3228 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:57:25.001835 3228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 23:57:25.001890 3228 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:57:25.003763 3228 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:57:25.003839 3228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:57:25.004108 3284 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap starting.
I20260928 23:57:25.015116 3228 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:35115
I20260928 23:57:25.015499 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:35115 every 8 connection(s)
I20260928 23:57:25.016611 3228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:25.025249 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3228
I20260928 23:57:25.025695 3284 log.cc:824] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:25.025727 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:41875
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=44119
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:25.036144 3284 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:57:25.036702 3284 tablet_bootstrap.cc:492] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Bootstrap complete.
I20260928 23:57:25.038293 3284 ts_tablet_manager.cc:1397] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.004s
I20260928 23:57:25.042006 3284 raft_consensus.cc:348] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:25.042634 3284 raft_consensus.cc:729] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770cce1c3e2442aa94bc53bb13ac7caf, State: Initialized, Role: FOLLOWER
I20260928 23:57:25.043228 3284 consensus_queue.cc:260] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:25.044453 3284 ts_tablet_manager.cc:1428] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260928 23:57:25.056900 3394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:57:25.019167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:50153)
0928 23:57:25.019812 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.019878 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:25.021012 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:25.056033 (+ 35021us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":177,"threads_started":1}
W20260928 23:57:25.057415 3392 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:46439 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:46439: Client connection negotiation failed: client connection to 127.20.88.62:46439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:57:25.058212 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.057940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:53599)
0928 23:57:25.058070 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.058135 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 23:57:25.058876 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.058678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:50427)
0928 23:57:25.058782 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.058824 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:57:25.059113 3392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:57:25.156303 3396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:25.156620 3396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:25.156667 3396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:25.161207 3396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:25.161329 3396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:25.161423 3396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:25.167372 3396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:41875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=44119
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:25.168695 3396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:25.169829 3396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:25.181164 3405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:25.181164 3403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:25.181145 3402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:25.181422 3396 server_base.cc:1034] running on GCE node
I20260928 23:57:25.182054 3396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:25.182752 3396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:25.183946 3396 hybrid_clock.cc:648] HybridClock initialized: now 1790639845183928 us; error 34 us; skew 500 ppm
Sep 28 23:57:25 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639845, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:25.191370 3396 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:25.192531 3396 webserver.cc:460] Webserver started at http://127.20.88.2:44119/ using document root <none> and password file <none>
I20260928 23:57:25.193193 3396 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:25.193295 3396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:25.197678 3396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:57:25.200090 3412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.201548 3396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:25.201733 3396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "539b4738f4994589b91c6ffca17107ea"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "4c87162e502def35a3a44b852ac7fad6"
server_key_iv: "33ffee6d625563e0545114a253609954"
server_key_version: "encryptionkey@0"
I20260928 23:57:25.202373 3396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:25.217344 3396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:25.220968 3396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:25.221196 3396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:25.221837 3396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:25.222723 3396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:25.222793 3396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.222867 3396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:25.222908 3396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.233506 3396 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:41875
I20260928 23:57:25.233604 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:41875 every 8 connection(s)
I20260928 23:57:25.234647 3396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:25.235070 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3396
I20260928 23:57:25.235602 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:34823
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:46439
--webserver_port=35517
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:57:25.239243 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.237239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:58719)
0928 23:57:25.237797 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.237912 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
W20260928 23:57:25.239789 3526 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:46439 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:46439: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
I20260928 23:57:25.240563 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.240326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:51885)
0928 23:57:25.240442 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.240506 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:57:25.241287 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.241042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:38095)
0928 23:57:25.241168 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.241220 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 23:57:25.241470 3526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:57:25.357802 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:25.358165 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:25.358254 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:25.362663 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:25.362764 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:25.362859 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:25.368675 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:34823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=35517
--tserver_master_addrs=127.20.88.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:25.369915 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:25.371048 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:25.381790 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:25.381783 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:25.381786 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:25.381842 3528 server_base.cc:1034] running on GCE node
I20260928 23:57:25.382423 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:25.383145 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:25.384322 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790639845384284 us; error 49 us; skew 500 ppm
Sep 28 23:57:25 dist-test-slave-wvpw krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639845, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:25.387245 3528 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:25.388330 3528 webserver.cc:460] Webserver started at http://127.20.88.3:35517/ using document root <none> and password file <none>
I20260928 23:57:25.388994 3528 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:25.389098 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:25.393846 3528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:57:25.396373 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.397887 3528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:25.398077 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "864e8b94de4f47b280f2b13e1a947e09"
format_stamp: "Formatted at 2026-09-28 23:57:22 on dist-test-slave-wvpw"
server_key: "8801ac5e9c8281ef2dcd4f8e18874cec"
server_key_iv: "3411e2126b56daea44954ec422445c35"
server_key_version: "encryptionkey@0"
I20260928 23:57:25.398744 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:25.412075 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:25.415916 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:25.416189 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:25.416988 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:25.417907 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:25.417986 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.418063 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:25.418115 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:25.427906 3528 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:34823
I20260928 23:57:25.427922 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:34823 every 8 connection(s)
I20260928 23:57:25.429068 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:25.433269 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.431275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:35093)
0928 23:57:25.431811 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.431937 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":109,"threads_started":1}
I20260928 23:57:25.433588 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3528
W20260928 23:57:25.433918 3658 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:46439 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:46439: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
I20260928 23:57:25.434752 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.434378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:34561)
0928 23:57:25.434593 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.434658 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 23:57:25.435386 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:25.435174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:33433)
0928 23:57:25.435288 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:25.435334 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:57:25.435693 3658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:57:26.060472 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.059852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:36789)
0928 23:57:26.060199 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:26.060351 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":84,"threads_started":1}
I20260928 23:57:26.242693 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.242200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:36835)
0928 23:57:26.242510 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:26.242578 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260928 23:57:26.436748 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.436338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:32945)
0928 23:57:26.436566 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:26.436618 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260928 23:57:26.714648 3665 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:26.714913 3665 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:26.716148 3665 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
Sep 28 23:57:26 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.3@KRBTEST.COM
Sep 28 23:57:26 dist-test-slave-wvpw krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639844, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.2@KRBTEST.COM
I20260928 23:57:26.731330 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.716458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34823 (local address 127.20.88.1:36557)
0928 23:57:26.716778 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:26.716793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:26.716907 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:26.718569 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:26.718575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:26.719138 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:26.719654 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:26.719670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.721092 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.721095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:26.722079 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:26.722089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.722880 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:26.723070 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:26.723893 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:26.723905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:26.725871 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:26.728509 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:26.728516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:26.728533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:26.728929 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:26.729310 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:26.729317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:26.729496 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:26.730195 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:26.730202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:26.730505 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 23:57:26.730687 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 23:57:26.731073 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":204,"spinlock_wait_cycles":13696,"thread_start_us":61,"threads_started":1}
I20260928 23:57:26.731454 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.716587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57919 (local address 127.20.88.2:41875)
0928 23:57:26.717048 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:26.717089 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:26.717127 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:57:26.717812 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:26.717817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:26.718174 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:26.718500 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:26.719847 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.720931 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722243 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.722717 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722880 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:26.726395 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:26.726400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:26.726411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:26.726443 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:26.728496 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729075 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:26.729081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:26.729087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:26.729214 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729599 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:26.729603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:26.729604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:26.730067 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:26.730222 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:26.730963 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 23:57:26.731262 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":80,"threads_started":1}
I20260928 23:57:26.731472 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.716667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:36557 (local address 127.20.88.3:34823)
0928 23:57:26.717117 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:26.717137 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:26.717170 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:57:26.717820 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:26.717825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:26.718174 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:26.718503 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:26.719851 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.720920 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722216 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.722725 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722916 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:26.726150 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:26.726159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:26.726177 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:26.726216 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:26.728373 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729055 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:26.729062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:26.729068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:26.729205 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729628 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:26.729630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:26.729632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:26.730066 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:26.730222 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:26.730942 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 23:57:26.731292 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":103,"threads_started":1}
I20260928 23:57:26.731385 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:26.716375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41875 (local address 127.20.88.1:57919)
0928 23:57:26.716734 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:26.716753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:26.716878 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:26.718569 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:26.718576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:26.719136 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:26.719654 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:26.719671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.721106 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.721109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:26.722079 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:26.722089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:26.722858 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:26.722863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:26.723071 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:26.723849 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:26.723868 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:26.726187 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:26.728633 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:26.728636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:26.728639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:26.728936 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:26.729342 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:26.729345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:26.729346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:26.729491 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:26.730195 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:26.730202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:26.730508 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 23:57:26.730812 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 23:57:26.731179 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":367,"spinlock_wait_cycles":32640,"thread_start_us":91,"threads_started":1}
W20260928 23:57:26.734638 3279 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
W20260928 23:57:26.734829 3278 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:26.734948 3278 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770cce1c3e2442aa94bc53bb13ac7caf; no voters: 539b4738f4994589b91c6ffca17107ea, 864e8b94de4f47b280f2b13e1a947e09
I20260928 23:57:26.735293 3665 raft_consensus.cc:2726] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:57:27.061617 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:27.061304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:47969)
0928 23:57:27.061462 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:27.061520 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 23:57:27.244105 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:27.243614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:56665)
0928 23:57:27.243921 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:27.243998 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260928 23:57:27.438037 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:27.437616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:60593)
0928 23:57:27.437877 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:27.437934 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260928 23:57:28.062995 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:28.062487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:36805)
0928 23:57:28.062806 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:28.062875 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260928 23:57:28.245656 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:28.245151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:55887)
0928 23:57:28.245448 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:28.245554 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260928 23:57:28.439378 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:28.438900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:35977)
0928 23:57:28.439217 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:28.439276 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":131,"threads_started":1}
I20260928 23:57:28.633770 3675 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:28.633967 3675 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:28.634495 3675 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
W20260928 23:57:28.635402 3279 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
W20260928 23:57:28.635499 3278 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:28.635618 3278 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770cce1c3e2442aa94bc53bb13ac7caf; no voters: 539b4738f4994589b91c6ffca17107ea, 864e8b94de4f47b280f2b13e1a947e09
I20260928 23:57:28.635849 3675 raft_consensus.cc:2726] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:57:29.064332 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:29.063844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:42639)
0928 23:57:29.064131 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:29.064192 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260928 23:57:29.247033 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:29.246593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:45393)
0928 23:57:29.246865 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:29.246928 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260928 23:57:29.440831 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:29.440276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:49487)
0928 23:57:29.440660 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:29.440734 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":97,"threads_started":1}
I20260928 23:57:30.065955 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:30.065461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:59019)
0928 23:57:30.065773 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:30.065834 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260928 23:57:30.248553 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:30.248037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:53795)
0928 23:57:30.248380 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:30.248444 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":82,"threads_started":1}
I20260928 23:57:30.442188 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:30.441734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:33315)
0928 23:57:30.442025 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:30.442090 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":161,"threads_started":1}
I20260928 23:57:31.067267 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:31.066818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:35409)
0928 23:57:31.067110 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:31.067169 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260928 23:57:31.250020 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:31.249529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:51103)
0928 23:57:31.249852 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:31.249919 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260928 23:57:31.419214 3684 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:31.419428 3684 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:31.419940 3684 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
W20260928 23:57:31.420841 3278 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
W20260928 23:57:31.420986 3279 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:31.421104 3279 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770cce1c3e2442aa94bc53bb13ac7caf; no voters: 539b4738f4994589b91c6ffca17107ea, 864e8b94de4f47b280f2b13e1a947e09
I20260928 23:57:31.421295 3684 raft_consensus.cc:2726] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:57:31.443504 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:31.443088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:50897)
0928 23:57:31.443352 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:31.443408 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260928 23:57:32.068712 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:32.068149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:51691)
0928 23:57:32.068513 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:32.068606 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":161,"threads_started":1}
I20260928 23:57:32.251569 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:32.251067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:45423)
0928 23:57:32.251396 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:32.251462 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":86,"threads_started":1}
I20260928 23:57:32.444888 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:32.444374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:36157)
0928 23:57:32.444727 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:32.444789 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260928 23:57:33.070000 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:33.069484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:37543)
0928 23:57:33.069813 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:33.069877 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
I20260928 23:57:33.253084 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:33.252625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:41781)
0928 23:57:33.252915 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:33.252980 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260928 23:57:33.446241 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:33.445774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:44737)
0928 23:57:33.446074 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:33.446136 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260928 23:57:34.071393 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:34.070922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:39583)
0928 23:57:34.071227 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:34.071291 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260928 23:57:34.254423 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:34.253985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:38903)
0928 23:57:34.254258 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:34.254323 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260928 23:57:34.447655 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:34.447206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:57063)
0928 23:57:34.447492 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:34.447557 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
I20260928 23:57:35.072835 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:35.072400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:50049)
0928 23:57:35.072675 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:35.072734 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260928 23:57:35.255981 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:35.255449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:32837)
0928 23:57:35.255810 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:35.255881 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260928 23:57:35.386577 3697 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:35.386726 3697 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:35.387190 3697 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
W20260928 23:57:35.388022 3278 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
W20260928 23:57:35.388128 3279 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:35.388195 3279 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770cce1c3e2442aa94bc53bb13ac7caf; no voters: 539b4738f4994589b91c6ffca17107ea, 864e8b94de4f47b280f2b13e1a947e09
I20260928 23:57:35.388389 3697 raft_consensus.cc:2726] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:57:35.449247 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:35.448696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:41279)
0928 23:57:35.449060 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:35.449139 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260928 23:57:36.074299 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:36.073840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:50783)
0928 23:57:36.074133 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:36.074199 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260928 23:57:36.257711 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:36.257161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:39745)
0928 23:57:36.257514 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:36.257583 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
I20260928 23:57:36.450749 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:36.450290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:59727)
0928 23:57:36.450579 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:36.450648 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260928 23:57:37.075513 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:37.075092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:41995)
0928 23:57:37.075323 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:37.075378 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260928 23:57:37.165405 3703 raft_consensus.cc:482] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:37.165552 3703 raft_consensus.cc:504] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539b4738f4994589b91c6ffca17107ea" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41875 } } peers { permanent_uuid: "864e8b94de4f47b280f2b13e1a947e09" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34823 } } peers { permanent_uuid: "770cce1c3e2442aa94bc53bb13ac7caf" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 35115 } }
I20260928 23:57:37.166095 3703 leader_election.cc:288] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875), 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823)
W20260928 23:57:37.166953 3278 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 539b4738f4994589b91c6ffca17107ea (127.20.88.2:41875): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
W20260928 23:57:37.167055 3279 leader_election.cc:341] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 864e8b94de4f47b280f2b13e1a947e09 (127.20.88.3:34823): Not found: Tablet not found: 7e03289cc820469a9a3b30edf11828a4
I20260928 23:57:37.167171 3279 leader_election.cc:302] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770cce1c3e2442aa94bc53bb13ac7caf; no voters: 539b4738f4994589b91c6ffca17107ea, 864e8b94de4f47b280f2b13e1a947e09
I20260928 23:57:37.167342 3703 raft_consensus.cc:2726] T 7e03289cc820469a9a3b30edf11828a4 P 770cce1c3e2442aa94bc53bb13ac7caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:57:37.259203 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:37.258732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:44427)
0928 23:57:37.259036 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:37.259103 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260928 23:57:37.452338 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:37.451787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:42399)
0928 23:57:37.452080 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:37.452148 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260928 23:57:38.076860 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:38.076415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:37165)
0928 23:57:38.076662 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:38.076719 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260928 23:57:38.260833 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:38.260307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:33219)
0928 23:57:38.260668 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:38.260732 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260928 23:57:38.453884 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:38.453299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:49563)
0928 23:57:38.453714 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:38.453785 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260928 23:57:39.078235 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:39.077799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:38015)
0928 23:57:39.078082 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:39.078137 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260928 23:57:39.262272 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:39.261801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:35281)
0928 23:57:39.262104 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:39.262170 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260928 23:57:39.455718 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:39.455085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:50845)
0928 23:57:39.455500 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:39.455560 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":111,"threads_started":1}
I20260928 23:57:40.079730 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:40.079274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.1:51991)
0928 23:57:40.079564 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:40.079628 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
I20260928 23:57:40.263774 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:40.263321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.2:60781)
0928 23:57:40.263609 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:40.263675 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260928 23:57:40.445147 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3228
I20260928 23:57:40.451411 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3396
I20260928 23:57:40.457523 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:40.456963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:46439 (local address 127.20.88.3:32855)
0928 23:57:40.457269 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:40.457335 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260928 23:57:40.457607 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3528
2026-09-28T23:57:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19543 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:57:40 dist-test-slave-wvpw krb5kdc[3717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:57:40 dist-test-slave-wvpw krb5kdc[3717](info): set up 2 sockets
Sep 28 23:57:40 dist-test-slave-wvpw krb5kdc[3717](info): commencing operation
krb5kdc: starting...
W20260928 23:57:41.517076 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:57:41 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:57:41.651618 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:41.769335 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:41.769681 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:41.769757 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:41.774024 3730 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:57:41.774145 3730 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:57:41.774174 3730 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:57:41.774241 3730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:57:41.774323 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:41.774387 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:41.774437 3730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:57:41.774472 3730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:57:41.780448 3730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:41.781678 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:41.782786 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:41.792881 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:41.792866 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:41.792945 3730 server_base.cc:1034] running on GCE node
W20260928 23:57:41.792874 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:41.793797 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:41.795269 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:41.796517 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790639861796479 us; error 56 us; skew 500 ppm
Sep 28 23:57:41 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639861, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:41.805856 3730 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:57:41.806914 3730 webserver.cc:460] Webserver started at http://127.20.88.62:41347/ using document root <none> and password file <none>
I20260928 23:57:41.807565 3730 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:41.807675 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:41.807914 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:41.810156 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "f1561f1870fb462d85456a14501198dd"
format_stamp: "Formatted at 2026-09-28 23:57:41 on dist-test-slave-wvpw"
server_key: "04082f165927937a731fb3c040b0b87a"
server_key_iv: "b1c80c52256e325820511eb79728b973"
server_key_version: "encryptionkey@0"
I20260928 23:57:41.810808 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "f1561f1870fb462d85456a14501198dd"
format_stamp: "Formatted at 2026-09-28 23:57:41 on dist-test-slave-wvpw"
server_key: "04082f165927937a731fb3c040b0b87a"
server_key_iv: "b1c80c52256e325820511eb79728b973"
server_key_version: "encryptionkey@0"
I20260928 23:57:41.814908 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 23:57:41.817688 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:41.819116 3730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:57:41.819259 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "f1561f1870fb462d85456a14501198dd"
format_stamp: "Formatted at 2026-09-28 23:57:41 on dist-test-slave-wvpw"
server_key: "04082f165927937a731fb3c040b0b87a"
server_key_iv: "b1c80c52256e325820511eb79728b973"
server_key_version: "encryptionkey@0"
I20260928 23:57:41.819388 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:41.839799 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:41.843472 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:41.843716 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:41.851138 3730 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35997
I20260928 23:57:41.851517 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35997 every 8 connection(s)
I20260928 23:57:41.852208 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:57:41.855258 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:41.859172 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3730
I20260928 23:57:41.859333 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:57:41.859627 20832 external_mini_cluster.cc:1428] Setting key 2e22053c730db950593599ea6a9a9250
I20260928 23:57:41.861724 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap starting.
I20260928 23:57:41.864605 3798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:41.865504 3798 log.cc:824] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:41.867851 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: No bootstrap required, opened a new log
Sep 28 23:57:41 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:41.871790 3798 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:57:41.872143 3798 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:41.872197 3798 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1561f1870fb462d85456a14501198dd, State: Initialized, Role: FOLLOWER
I20260928 23:57:41.872747 3798 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:57:41.872931 3798 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:57:41.873001 3798 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:57:41.873098 3798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:41.874356 3798 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:57:41.874752 3798 leader_election.cc:302] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1561f1870fb462d85456a14501198dd; no voters:
I20260928 23:57:41.875162 3798 leader_election.cc:288] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:57:41.875375 3803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:41.875633 3803 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 LEADER]: Becoming Leader. State: Replica: f1561f1870fb462d85456a14501198dd, State: Running, Role: LEADER
I20260928 23:57:41.876015 3803 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:57:41.876374 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:41.861182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.20.88.62:35997)
0928 23:57:41.861650 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:41.861669 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:41.861712 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:57:41.862583 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:41.862607 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:41.863056 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:41.863448 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:41.864098 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:41.865650 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:41.866578 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:41.867246 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:41.867419 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:41.870061 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:41.870068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:41.870087 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:41.870124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:41.872796 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:41.873393 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:41.873399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:41.873406 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:41.873547 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:41.873858 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:41.873861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:41.873863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:41.874455 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:41.874645 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:41.874975 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 23:57:41.875340 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":127,"threads_started":1}
I20260928 23:57:41.876663 3798 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:57:41.877825 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1561f1870fb462d85456a14501198dd. Latest consensus state: current_term: 1 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:57:41.877930 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:57:41.877971 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:41.877995 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:57:41.878973 3812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:57:41.882568 3812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:57:41.888226 3812 catalog_manager.cc:1347] Generated new cluster ID: 7da02be5318c4eb38e2f9c03810c6485
I20260928 23:57:41.888351 3812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:57:41.914126 3812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:57:41.915654 3812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:57:41.922190 3812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Generated new TSK 0
I20260928 23:57:41.923204 3812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:57:41.963338 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:42.083577 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:42.083873 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:42.083917 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:42.088088 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:42.088174 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:42.088307 3824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:57:42.094103 3824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:42.095235 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:42.096380 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:42.106776 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:42.106777 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:42.106777 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:42.107120 3824 server_base.cc:1034] running on GCE node
I20260928 23:57:42.107591 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:42.108296 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:42.109508 3824 hybrid_clock.cc:648] HybridClock initialized: now 1790639862109493 us; error 31 us; skew 500 ppm
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:42.112561 3824 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:57:42.113564 3824 webserver.cc:460] Webserver started at http://127.20.88.1:38195/ using document root <none> and password file <none>
I20260928 23:57:42.114208 3824 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:42.114312 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:42.114570 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:42.117036 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "2de13984a4fb0c2cc8cfd72330321692"
server_key_iv: "4e7361c2be2c8de94d18aa284b33cda9"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.117682 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "2de13984a4fb0c2cc8cfd72330321692"
server_key_iv: "4e7361c2be2c8de94d18aa284b33cda9"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.123559 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 23:57:42.127629 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.129242 3824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:42.129379 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "2de13984a4fb0c2cc8cfd72330321692"
server_key_iv: "4e7361c2be2c8de94d18aa284b33cda9"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.129513 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:42.147639 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:42.151377 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:42.151618 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:42.152403 3824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:42.153335 3824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:42.153383 3824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.153425 3824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:42.153476 3824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.163089 3824 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:44005
I20260928 23:57:42.163151 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:44005 every 8 connection(s)
I20260928 23:57:42.164122 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:57:42.171783 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3824
I20260928 23:57:42.171962 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:57:42.172317 20832 external_mini_cluster.cc:1428] Setting key 07cb13ae8ed12606e2e5fd091a183cb8
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:42.179832 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.166222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:53691 (local address 127.20.88.62:35997)
0928 23:57:42.166381 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.166385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.167297 (+ 912us) server_negotiation.cc:407] Connection header received
0928 23:57:42.168860 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.168864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.168925 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.169039 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:42.170337 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.170895 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.172228 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.172568 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.172693 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.175599 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:42.175603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:42.175606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:42.175641 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:42.177360 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.178137 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.178141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.178143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.178239 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.178615 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.178620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.178622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.178836 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:42.178967 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.179510 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.179659 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:57:42.180824 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.166644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:53691)
0928 23:57:42.167125 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.167172 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.168616 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.169233 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.169241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.169637 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:42.170160 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.170176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.171052 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.171055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.172095 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.172107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.172755 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.172759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.172944 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.173636 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:42.173654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:42.175385 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:42.177601 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.177609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.177630 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.178024 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.178345 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.178348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.178350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.178513 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.178954 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:42.178963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:42.179249 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.179463 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.179829 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:57:42.182229 3953 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:57:42.182530 3953 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:42.183045 3953 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:57:42.184906 3760 ts_manager.cc:194] Registered new tserver with Master: 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:57:42.186101 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:53691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:57:42.203722 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:42.349329 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:42.349650 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:42.349768 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:42.354064 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:42.354173 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:42.354271 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:57:42.360029 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:42.361331 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:42.362488 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:42.375030 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:42.375089 3958 server_base.cc:1034] running on GCE node
W20260928 23:57:42.375197 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:42.375031 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:42.375907 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:42.376772 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:42.378049 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790639862378009 us; error 71 us; skew 500 ppm
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:42.381662 3958 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:57:42.383160 3958 webserver.cc:460] Webserver started at http://127.20.88.2:44329/ using document root <none> and password file <none>
I20260928 23:57:42.384030 3958 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:42.384136 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:42.384457 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:42.388041 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "d588be444113528a242fdebc328b793f"
server_key_iv: "33d9c30fcf3b10e6d803e7934aa4b816"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.389029 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "d588be444113528a242fdebc328b793f"
server_key_iv: "33d9c30fcf3b10e6d803e7934aa4b816"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.395284 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 23:57:42.399355 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.401414 3958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:57:42.401592 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "d588be444113528a242fdebc328b793f"
server_key_iv: "33d9c30fcf3b10e6d803e7934aa4b816"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.401772 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:42.422581 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:42.427256 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:42.427512 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:42.428462 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:42.429872 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:42.429948 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.430032 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:42.430071 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.444835 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:38183 every 8 connection(s)
I20260928 23:57:42.444836 3958 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:38183
I20260928 23:57:42.445937 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:57:42.452041 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 3958
I20260928 23:57:42.452150 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:57:42.452491 20832 external_mini_cluster.cc:1428] Setting key ffa2946e6b3978a00e05f49618a15315
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:57:42.463667 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.448578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:51383 (local address 127.20.88.62:35997)
0928 23:57:42.448763 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.448768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.449806 (+ 1038us) server_negotiation.cc:407] Connection header received
0928 23:57:42.451584 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.451587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.451643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.451728 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:42.453409 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.454113 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.455498 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.455955 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.456115 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.458938 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:42.458946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:42.458954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:42.458999 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:42.460976 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.461666 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.461673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.461678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.461791 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.462201 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.462208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.462211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.462532 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:42.462654 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.463253 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.463436 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 23:57:42.464610 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.449024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:51383)
0928 23:57:42.449601 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.449667 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.451308 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.451899 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.451906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.452469 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:42.453228 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.453248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.454288 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.454291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.455336 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.455347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.456172 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.456175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.456466 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.457134 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:42.457151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:42.458712 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:42.461145 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.461151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.461168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.461511 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.461906 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.461908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.461910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.462068 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.462675 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:42.462681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:42.462969 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.463138 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.463488 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":133,"threads_started":1}
I20260928 23:57:42.466053 4087 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:57:42.466310 4087 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:57:42.467042 4087 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
I20260928 23:57:42.468566 3760 ts_manager.cc:194] Registered new tserver with Master: 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183)
I20260928 23:57:42.469224 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:51383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:57:42.488173 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:57:42.613272 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:57:42.613603 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:57:42.613685 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:57:42.618165 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:57:42.618255 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:57:42.618321 4092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:57:42.624154 4092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:57:42.625393 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:57:42.626502 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:57:42.636979 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:57:42.636968 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:42.636974 4092 server_base.cc:1034] running on GCE node
W20260928 23:57:42.637005 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:57:42.637573 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:57:42.638286 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:57:42.639483 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790639862639463 us; error 32 us; skew 500 ppm
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:57:42.642588 4092 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:57:42.643725 4092 webserver.cc:460] Webserver started at http://127.20.88.3:34711/ using document root <none> and password file <none>
I20260928 23:57:42.644405 4092 fs_manager.cc:359] Metadata directory not provided
I20260928 23:57:42.644481 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:57:42.644663 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:57:42.646920 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "aea2317bbecb414297abec899caf8d40"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "704ae11508ad92fa56ca13e0945555ea"
server_key_iv: "949f98751d7e04c67841b7200e297158"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.647780 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "aea2317bbecb414297abec899caf8d40"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "704ae11508ad92fa56ca13e0945555ea"
server_key_iv: "949f98751d7e04c67841b7200e297158"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.652040 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:57:42.654837 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.656421 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:57:42.656579 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "aea2317bbecb414297abec899caf8d40"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "704ae11508ad92fa56ca13e0945555ea"
server_key_iv: "949f98751d7e04c67841b7200e297158"
server_key_version: "encryptionkey@0"
I20260928 23:57:42.656711 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:57:42.672657 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:57:42.676332 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:57:42.676559 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:57:42.677263 4092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:57:42.678292 4092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:57:42.678367 4092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.678453 4092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:57:42.678522 4092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:57:42.689073 4092 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:35045
I20260928 23:57:42.689108 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:35045 every 8 connection(s)
I20260928 23:57:42.690097 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:57:42.696130 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4092
I20260928 23:57:42.696302 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:57:42.696676 20832 external_mini_cluster.cc:1428] Setting key 5a60cb3f2287b8d07ce039cabe7f7fc0
Sep 28 23:57:42 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639862, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:57:42.706441 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.692106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:59343 (local address 127.20.88.62:35997)
0928 23:57:42.692341 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.692346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.693450 (+ 1104us) server_negotiation.cc:407] Connection header received
0928 23:57:42.694888 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.694891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.694950 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.695052 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:42.696530 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.697144 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.699456 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.699681 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.699786 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.702588 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:42.702592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:42.702594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:42.702632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:42.704325 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.704877 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.704879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.704881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.704963 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.705298 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.705301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.705302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.705536 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:42.705616 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.706162 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.706277 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:57:42.707636 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.692747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:59343)
0928 23:57:42.693278 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.693324 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.694613 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.695228 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.695237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.695677 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:57:42.696304 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.696331 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.697370 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.697400 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.699254 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.699267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.699890 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.699893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.700053 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.700835 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:57:42.700853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:57:42.702369 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:57:42.704440 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.704446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.704461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.704751 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.705046 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:57:42.705048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:57:42.705049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:57:42.705197 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:57:42.705631 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:57:42.705637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:57:42.705930 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.706096 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.706392 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":116,"threads_started":1}
I20260928 23:57:42.709129 4221 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:57:42.709487 4221 heartbeater.cc:464] Registering TS with master...
I20260928 23:57:42.710053 4221 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
I20260928 23:57:42.711241 3762 ts_manager.cc:194] Registered new tserver with Master: aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
I20260928 23:57:42.711910 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:59343
I20260928 23:57:42.723892 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:57:42.734453 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.726275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.20.88.62:35997)
0928 23:57:42.726419 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.726423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.726644 (+ 221us) server_negotiation.cc:407] Connection header received
0928 23:57:42.726724 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.726726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.726774 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.726855 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:57:42.727550 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.728070 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.728973 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.729195 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.729279 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.730782 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:57:42.730786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:57:42.730788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:57:42.730816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:57:42.732463 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.732999 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.733004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.733031 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.733132 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:57:42.733500 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:57:42.733503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:57:42.733504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:57:42.733750 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:57:42.733865 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.734143 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.734280 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:57:42.738137 3762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:57:42.741573 3762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:57:42.756407 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.750933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.20.88.3:35045)
0928 23:57:42.751574 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.751587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.751610 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:57:42.751690 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.751696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.751919 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.752093 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:42.752568 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753332 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.754971 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.755693 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.755823 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.756006 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.756115 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.756213 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":182,"threads_started":1}
I20260928 23:57:42.756409 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.750214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:35045 (local address 127.0.0.1:40540)
0928 23:57:42.750725 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.750776 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.750924 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.752281 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.752284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.752301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:42.752451 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.752455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753469 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.753474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.754859 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.754870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.755823 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.755826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.755933 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.755971 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.756088 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.756215 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":72,"thread_start_us":98,"threads_started":1}
I20260928 23:57:42.756655 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.750962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.0.0.1:33938)
0928 23:57:42.751275 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.751287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.751368 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.752083 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.752089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.752113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:42.752298 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.752305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753905 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.753908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.755102 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.755110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.756249 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.756253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.756347 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.756371 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.756426 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.756498 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":78,"thread_start_us":90,"threads_started":1}
I20260928 23:57:42.756814 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.750410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38183 (local address 127.0.0.1:35532)
0928 23:57:42.750974 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.750988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.751086 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.752292 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.752295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.752311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:42.752470 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.752476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753945 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.753948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.755191 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.755202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.756449 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.756453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.756538 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.756558 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.756623 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.756689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":105,"thread_start_us":110,"threads_started":1}
I20260928 23:57:42.757287 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.750582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.20.88.2:38183)
0928 23:57:42.751703 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.751713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.751732 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:57:42.751796 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.751800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.751958 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.752101 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:42.752601 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753775 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.755318 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.756095 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.756201 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.756391 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.757067 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.757168 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":114,"threads_started":1}
I20260928 23:57:42.757689 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.751281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.20.88.1:44005)
0928 23:57:42.751606 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.751613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.751634 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:57:42.751711 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.751716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.751909 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.752464 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:42.752603 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.753775 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.755227 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.756078 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.757231 (+ 1153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.757377 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.757445 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.757544 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260928 23:57:42.760273 4155 tablet_service.cc:1467] Processing CreateTablet for tablet 5d87763b201840f0b09fce28cf2faa63 (DEFAULT_TABLE table=test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:42.760273 4021 tablet_service.cc:1467] Processing CreateTablet for tablet 5d87763b201840f0b09fce28cf2faa63 (DEFAULT_TABLE table=test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:42.760270 3887 tablet_service.cc:1467] Processing CreateTablet for tablet 5d87763b201840f0b09fce28cf2faa63 (DEFAULT_TABLE table=test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:57:42.761336 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d87763b201840f0b09fce28cf2faa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:42.761354 4155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d87763b201840f0b09fce28cf2faa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:42.761349 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d87763b201840f0b09fce28cf2faa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:57:42.767827 4238 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap starting.
I20260928 23:57:42.769565 4239 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Bootstrap starting.
I20260928 23:57:42.770293 4238 tablet_bootstrap.cc:654] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:42.771051 4238 log.cc:824] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:42.771490 4239 tablet_bootstrap.cc:654] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:42.771777 4240 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Bootstrap starting.
I20260928 23:57:42.772861 4239 log.cc:824] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:42.773432 4238 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: No bootstrap required, opened a new log
I20260928 23:57:42.773723 4238 ts_tablet_manager.cc:1397] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:57:42.774412 4240 tablet_bootstrap.cc:654] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Neither blocks nor log segments found. Creating new log.
I20260928 23:57:42.774984 4239 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: No bootstrap required, opened a new log
I20260928 23:57:42.775208 4239 ts_tablet_manager.cc:1397] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:57:42.775184 4240 log.cc:824] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:57:42.777060 4240 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: No bootstrap required, opened a new log
I20260928 23:57:42.777276 4240 ts_tablet_manager.cc:1397] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:57:42.778646 4238 raft_consensus.cc:348] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.778954 4238 raft_consensus.cc:374] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:42.779026 4238 raft_consensus.cc:729] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f97c6ebe1e4f6997b9001536f6e0ea, State: Initialized, Role: FOLLOWER
I20260928 23:57:42.779311 4239 raft_consensus.cc:348] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.779578 4239 raft_consensus.cc:374] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:42.779626 4239 raft_consensus.cc:729] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea2317bbecb414297abec899caf8d40, State: Initialized, Role: FOLLOWER
I20260928 23:57:42.779645 4238 consensus_queue.cc:260] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.780100 4239 consensus_queue.cc:260] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.780774 4238 ts_tablet_manager.cc:1428] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:57:42.781016 4239 ts_tablet_manager.cc:1428] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:57:42.781201 3953 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
I20260928 23:57:42.781327 4221 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
I20260928 23:57:42.781002 4240 raft_consensus.cc:348] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.782240 4240 raft_consensus.cc:374] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:57:42.782305 4240 raft_consensus.cc:729] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ec400d8f0e41cbb7cb9b23e356d9e7, State: Initialized, Role: FOLLOWER
I20260928 23:57:42.782763 4240 consensus_queue.cc:260] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.783429 4087 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
I20260928 23:57:42.783706 4240 ts_tablet_manager.cc:1428] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:57:42.803370 4245 raft_consensus.cc:482] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:57:42.803679 4245 raft_consensus.cc:504] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.804992 4245 leader_election.cc:288] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:57:42.809334 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.805349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53749 (local address 127.20.88.2:38183)
0928 23:57:42.805533 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.805538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.805560 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:57:42.805625 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.805630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.805682 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.805774 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:42.806086 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.806752 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808114 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.808655 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808808 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.808914 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.809098 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.809187 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:57:42.809417 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.805216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38183 (local address 127.20.88.3:53749)
0928 23:57:42.805366 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.805400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.805508 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.805795 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.805797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.805814 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:42.805971 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.805979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.806883 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.806886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.807982 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.807993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.808793 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.808934 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.808986 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.809067 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.809180 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":43}
I20260928 23:57:42.809475 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.805428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:39775 (local address 127.20.88.1:44005)
0928 23:57:42.805561 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.805565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.805902 (+ 337us) server_negotiation.cc:407] Connection header received
0928 23:57:42.805982 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.805985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.806046 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.806131 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:57:42.806445 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.807063 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808283 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.808810 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808934 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.809000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.809272 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.809360 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:57:42.809875 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.805229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.20.88.3:39775)
0928 23:57:42.805781 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.805797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:57:42.805895 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:57:42.806154 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:57:42.806157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:57:42.806176 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:57:42.806292 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.806298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.807182 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.807185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.808064 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:57:42.808078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.808940 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.808951 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:57:42.809136 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.809159 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:57:42.809234 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:57:42.809303 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":211,"thread_start_us":124,"threads_started":1}
I20260928 23:57:42.810261 4041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d87763b201840f0b09fce28cf2faa63" candidate_uuid: "aea2317bbecb414297abec899caf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" is_pre_election: true
I20260928 23:57:42.810599 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d87763b201840f0b09fce28cf2faa63" candidate_uuid: "aea2317bbecb414297abec899caf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" is_pre_election: true
I20260928 23:57:42.810657 4041 raft_consensus.cc:2454] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea2317bbecb414297abec899caf8d40 in term 0.
I20260928 23:57:42.810895 3907 raft_consensus.cc:2454] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea2317bbecb414297abec899caf8d40 in term 0.
I20260928 23:57:42.811213 4108 leader_election.cc:302] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40; no voters:
I20260928 23:57:42.811600 4245 raft_consensus.cc:2781] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:57:42.811721 4245 raft_consensus.cc:482] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:57:42.811781 4245 raft_consensus.cc:3037] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:42.813089 4245 raft_consensus.cc:504] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.813522 4245 leader_election.cc:288] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [CANDIDATE]: Term 1 election: Requested vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:57:42.813987 4041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d87763b201840f0b09fce28cf2faa63" candidate_uuid: "aea2317bbecb414297abec899caf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
I20260928 23:57:42.813987 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d87763b201840f0b09fce28cf2faa63" candidate_uuid: "aea2317bbecb414297abec899caf8d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
I20260928 23:57:42.814139 4041 raft_consensus.cc:3037] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:42.814139 3907 raft_consensus.cc:3037] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:57:42.815553 4041 raft_consensus.cc:2454] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea2317bbecb414297abec899caf8d40 in term 1.
I20260928 23:57:42.815553 3907 raft_consensus.cc:2454] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea2317bbecb414297abec899caf8d40 in term 1.
I20260928 23:57:42.815909 4110 leader_election.cc:302] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea, aea2317bbecb414297abec899caf8d40; no voters:
I20260928 23:57:42.816124 4245 raft_consensus.cc:2781] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:57:42.816418 4245 raft_consensus.cc:686] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 1 LEADER]: Becoming Leader. State: Replica: aea2317bbecb414297abec899caf8d40, State: Running, Role: LEADER
I20260928 23:57:42.816839 4245 consensus_queue.cc:237] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:57:42.820390 3760 catalog_manager.cc:5495] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea2317bbecb414297abec899caf8d40 (127.20.88.3). New cstate: current_term: 1 leader_uuid: "aea2317bbecb414297abec899caf8d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20260928 23:57:42.837020 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.833270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.20.88.3:35045)
0928 23:57:42.833404 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:57:42.833409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:57:42.833433 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:57:42.833580 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:57:42.833582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:57:42.833639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:57:42.833727 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:57:42.833953 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.834509 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.835396 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:57:42.835621 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:57:42.835721 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:57:42.836026 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:57:42.836541 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:57:42.836637 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:57:42.836723 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:57:42.836780 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:57:42.843807 3907 raft_consensus.cc:1261] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Refusing update from remote peer aea2317bbecb414297abec899caf8d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:42.843807 4041 raft_consensus.cc:1261] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Refusing update from remote peer aea2317bbecb414297abec899caf8d40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:57:42.844867 4248 consensus_queue.cc:1035] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:42.845137 4245 consensus_queue.cc:1035] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:57:42.860845 4253 mvcc.cc:205] Tried to move back new op lower bound from 7334460878199287808 to 7334460878100799488. Current Snapshot: MvccSnapshot[applied={T|T < 7334460878199287808}]
I20260928 23:57:42.870729 3760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:42.870904 3760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:57:42.873657 3760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Sending DeleteTablet for 3 replicas of tablet 5d87763b201840f0b09fce28cf2faa63
I20260928 23:57:42.874648 4155 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d87763b201840f0b09fce28cf2faa63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:40540
I20260928 23:57:42.874650 3887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d87763b201840f0b09fce28cf2faa63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:33938
I20260928 23:57:42.874650 4021 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d87763b201840f0b09fce28cf2faa63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:35532
I20260928 23:57:42.875375 4262 tablet_replica.cc:330] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: stopping tablet replica
I20260928 23:57:42.875598 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3824
I20260928 23:57:42.875628 4261 tablet_replica.cc:330] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: stopping tablet replica
I20260928 23:57:42.875813 4262 raft_consensus.cc:2229] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:57:42.875948 4261 raft_consensus.cc:2229] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:57:42.876348 4262 raft_consensus.cc:2258] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:42.876345 4261 raft_consensus.cc:2258] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:57:42.878772 4261 ts_tablet_manager.cc:1905] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:42.881702 4261 ts_tablet_manager.cc:1918] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:42.881827 4261 log.cc:1196] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/5d87763b201840f0b09fce28cf2faa63
I20260928 23:57:42.882165 4261 ts_tablet_manager.cc:1939] T 5d87763b201840f0b09fce28cf2faa63 P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting consensus metadata
I20260928 23:57:42.882308 4262 ts_tablet_manager.cc:1905] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:57:42.883648 3746 catalog_manager.cc:4841] TS 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): tablet 5d87763b201840f0b09fce28cf2faa63 (table test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]) successfully deleted
W20260928 23:57:42.883790 3748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:44005 (error 108)
I20260928 23:57:42.884703 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3958
I20260928 23:57:42.886816 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:57:42.886475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.0.0.1:33944)
0928 23:57:42.886619 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:57:42.886708 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:57:42.887044 3748 catalog_manager.cc:4584] TS 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d87763b201840f0b09fce28cf2faa63: Network error: Client connection negotiation failed: client connection to 127.20.88.1:44005: connect: Connection refused (error 111)
I20260928 23:57:42.887486 4262 ts_tablet_manager.cc:1918] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:57:42.887640 4262 log.cc:1196] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/5d87763b201840f0b09fce28cf2faa63
I20260928 23:57:42.888017 4262 ts_tablet_manager.cc:1939] T 5d87763b201840f0b09fce28cf2faa63 P aea2317bbecb414297abec899caf8d40: Deleting consensus metadata
I20260928 23:57:42.889266 3748 catalog_manager.cc:4841] TS aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): tablet 5d87763b201840f0b09fce28cf2faa63 (table test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]) successfully deleted
I20260928 23:57:42.892397 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4092
I20260928 23:57:42.898491 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 3730
I20260928 23:58:02.906029 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--webserver_interface=127.20.88.62
--webserver_port=41347
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:03.022358 4264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:03.022665 4264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:03.022710 4264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:03.027172 4264 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:58:03.027241 4264 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:58:03.027258 4264 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:58:03.027273 4264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:03.027292 4264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:03.027354 4264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:03.027384 4264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:03.027421 4264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:03.033113 4264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:03.034330 4264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:03.035456 4264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:03.049616 4271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.049674 4273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.049616 4270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:03.050055 4264 server_base.cc:1034] running on GCE node
I20260928 23:58:03.050908 4264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:03.052506 4264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:03.053756 4264 hybrid_clock.cc:648] HybridClock initialized: now 1790639883053727 us; error 64 us; skew 500 ppm
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:03.078644 4264 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:58:03.079767 4264 webserver.cc:460] Webserver started at http://127.20.88.62:41347/ using document root <none> and password file <none>
I20260928 23:58:03.080515 4264 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:03.080605 4264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:03.084928 4264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:03.087512 4280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.089438 4264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:03.089672 4264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "f1561f1870fb462d85456a14501198dd"
format_stamp: "Formatted at 2026-09-28 23:57:41 on dist-test-slave-wvpw"
server_key: "04082f165927937a731fb3c040b0b87a"
server_key_iv: "b1c80c52256e325820511eb79728b973"
server_key_version: "encryptionkey@0"
I20260928 23:58:03.090624 4264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:03.105906 4264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:03.109305 4264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:03.109525 4264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:03.117425 4264 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35997
I20260928 23:58:03.117808 4332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35997 every 8 connection(s)
I20260928 23:58:03.118744 4264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:03.123759 4333 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:58:03.123889 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4264
I20260928 23:58:03.124449 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:44005
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=38195
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:58:03.127386 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap starting.
I20260928 23:58:03.146065 4333 log.cc:824] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:03.155486 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:58:03.155972 4333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap complete.
I20260928 23:58:03.160396 4333 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:03.160847 4333 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1561f1870fb462d85456a14501198dd, State: Initialized, Role: FOLLOWER
I20260928 23:58:03.161423 4333 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:03.161559 4333 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:03.161660 4333 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:03.161778 4333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:58:03.163898 4333 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:03.164325 4333 leader_election.cc:302] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1561f1870fb462d85456a14501198dd; no voters:
I20260928 23:58:03.164695 4333 leader_election.cc:288] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:58:03.164791 4337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:58:03.164991 4337 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 LEADER]: Becoming Leader. State: Replica: f1561f1870fb462d85456a14501198dd, State: Running, Role: LEADER
I20260928 23:58:03.165364 4337 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:03.166028 4333 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:03.166781 4338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:58:03.166913 4338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:03.167830 4339 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1561f1870fb462d85456a14501198dd. Latest consensus state: current_term: 2 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:58:03.167939 4339 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:03.168210 4345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:03.173185 4345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:03.173868 4345 catalog_manager.cc:1259] Loaded cluster ID: 7da02be5318c4eb38e2f9c03810c6485
I20260928 23:58:03.173930 4345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:03.176498 4345 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:03.177223 4345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Loaded TSK: 0
I20260928 23:58:03.193065 4345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Generated new TSK 1
I20260928 23:58:03.193387 4345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:58:03.246990 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:03.247328 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:03.247426 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:03.252007 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:03.252111 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:03.252207 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:03.258239 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:44005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=38195
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:03.259537 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:03.260736 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:03.273010 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:03.273274 4335 server_base.cc:1034] running on GCE node
W20260928 23:58:03.273113 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.273011 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:03.274041 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:03.274935 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:03.276152 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790639883276127 us; error 34 us; skew 500 ppm
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:03.279984 4335 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:58:03.281309 4335 webserver.cc:460] Webserver started at http://127.20.88.1:38195/ using document root <none> and password file <none>
I20260928 23:58:03.282058 4335 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:03.282187 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:03.287715 4335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:03.290915 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.292730 4335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:58:03.292906 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "2de13984a4fb0c2cc8cfd72330321692"
server_key_iv: "4e7361c2be2c8de94d18aa284b33cda9"
server_key_version: "encryptionkey@0"
I20260928 23:58:03.293721 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:03.308342 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:03.312156 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:03.312404 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:03.313118 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:03.314484 4377 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:58:03.319317 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:58:03.319427 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 23:58:03.319500 4335 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:58:03.321956 4335 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:58:03.322032 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:03.322511 4377 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap starting.
I20260928 23:58:03.333094 4335 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:44005
I20260928 23:58:03.333586 4484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:44005 every 8 connection(s)
I20260928 23:58:03.334357 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:03.342695 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4335
I20260928 23:58:03.343314 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:38183
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=44329
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:58:03.345655 4377 log.cc:824] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:03.354364 4377 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:58:03.354872 4377 tablet_bootstrap.cc:492] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap complete.
I20260928 23:58:03.356411 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.336980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:49069 (local address 127.20.88.62:35997)
0928 23:58:03.337361 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.337382 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.337909 (+ 527us) server_negotiation.cc:407] Connection header received
0928 23:58:03.339704 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.339720 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.340157 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.340578 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.342347 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.343553 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.345133 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.345889 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.346103 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.349449 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.349458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.349482 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.349531 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.352116 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.352850 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.352861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.352869 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.353065 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.353511 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.353518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.353521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.354130 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.354296 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.355118 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.355493 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":129,"threads_started":1}
I20260928 23:58:03.356292 4377 ts_tablet_manager.cc:1397] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.004s
I20260928 23:58:03.356325 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.337043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:49069)
0928 23:58:03.337717 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.337777 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.339151 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.340689 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.340698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.341343 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:03.342126 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.342150 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.343767 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.343770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.344891 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.344908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.346067 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.346074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.346316 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.347065 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:03.347088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:03.349163 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:03.352306 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.352312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.352326 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.352696 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.353199 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.353202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.353204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.353380 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.354270 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:03.354277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:03.354585 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.354857 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.355220 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":161,"threads_started":1}
I20260928 23:58:03.359416 4485 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:58:03.359678 4485 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:03.359627 4377 raft_consensus.cc:348] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.360160 4377 raft_consensus.cc:729] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f97c6ebe1e4f6997b9001536f6e0ea, State: Initialized, Role: FOLLOWER
I20260928 23:58:03.360327 4485 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
I20260928 23:58:03.360714 4377 consensus_queue.cc:260] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.361754 4377 ts_tablet_manager.cc:1428] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:58:03.364744 4295 ts_manager.cc:194] Registered new tserver with Master: 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:58:03.368775 4295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:49069
I20260928 23:58:03.371665 4485 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.1@KRBTEST.COM
I20260928 23:58:03.380208 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.368032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.0.0.1:37734)
0928 23:58:03.368636 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.368699 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.368862 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.369695 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.369702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.370175 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:03.370320 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.370329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.371641 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.371644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.372652 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.372665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.373526 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.373530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.373649 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.374258 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:03.374274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:03.375999 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:03.378161 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.378173 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.378179 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.378524 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.378851 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.378854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.378856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.378951 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.379498 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:03.379504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:03.379645 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.379795 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.380012 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":218,"threads_started":1}
I20260928 23:58:03.380206 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.368159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.20.88.1:44005)
0928 23:58:03.369164 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.369177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.369205 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:58:03.369288 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.369293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.369510 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.369671 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.370470 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.371502 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.372798 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.373349 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.373481 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.376176 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.376187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.376201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.376322 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.378030 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.378654 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.378659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.378660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.378746 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.379069 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.379072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.379074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.379383 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.379496 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.379901 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.380051 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":228,"threads_started":1}
I20260928 23:58:03.381403 4419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d87763b201840f0b09fce28cf2faa63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:57:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:37734
I20260928 23:58:03.382429 4494 tablet_replica.cc:330] stopping tablet replica
I20260928 23:58:03.382771 4494 raft_consensus.cc:2229] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:03.383038 4494 raft_consensus.cc:2258] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:03.384064 4494 ts_tablet_manager.cc:1905] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:03.388164 4494 ts_tablet_manager.cc:1918] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:03.388461 4494 log.cc:1196] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal/wals/5d87763b201840f0b09fce28cf2faa63
I20260928 23:58:03.388763 4494 ts_tablet_manager.cc:1939] T 5d87763b201840f0b09fce28cf2faa63 P 67f97c6ebe1e4f6997b9001536f6e0ea: Deleting consensus metadata
I20260928 23:58:03.390683 4283 catalog_manager.cc:4841] TS 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005): tablet 5d87763b201840f0b09fce28cf2faa63 (table test-table [id=a02d2f1807c048cc971e4e5f79cd7d1f]) successfully deleted
W20260928 23:58:03.470531 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:03.470862 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:03.470948 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:03.475251 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:03.475375 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:03.475478 4489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:03.481392 4489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:38183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=44329
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:03.482587 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:03.483685 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:03.494504 4500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.494434 4502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.494436 4499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:03.494652 4489 server_base.cc:1034] running on GCE node
I20260928 23:58:03.495352 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:03.496095 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:03.497285 4489 hybrid_clock.cc:648] HybridClock initialized: now 1790639883497246 us; error 50 us; skew 500 ppm
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:03.500308 4489 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:58:03.501370 4489 webserver.cc:460] Webserver started at http://127.20.88.2:44329/ using document root <none> and password file <none>
I20260928 23:58:03.502017 4489 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:03.502138 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:03.506557 4489 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:58:03.509060 4509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.510406 4489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:03.510550 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "d588be444113528a242fdebc328b793f"
server_key_iv: "33d9c30fcf3b10e6d803e7934aa4b816"
server_key_version: "encryptionkey@0"
I20260928 23:58:03.511139 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:03.533862 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:03.537539 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:03.537847 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:03.538527 4489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:03.539465 4489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:03.539546 4489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.539623 4489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:03.539667 4489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.549208 4489 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:38183
I20260928 23:58:03.549237 4622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:38183 every 8 connection(s)
I20260928 23:58:03.550308 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:03.551709 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4489
I20260928 23:58:03.552175 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:35045
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=34711
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:03.568869 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.552415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:37111 (local address 127.20.88.62:35997)
0928 23:58:03.552580 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.552587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.553627 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 23:58:03.555711 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.555717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.555793 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.555855 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.558177 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.559078 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.561152 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.561599 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.561731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.565007 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.565011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.565013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.565051 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.566562 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.567120 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.567124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.567125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.567212 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.567588 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.567594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.567597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.567802 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.567908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.568513 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.568658 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:58:03.569705 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.552864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:37111)
0928 23:58:03.553433 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.553500 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.555375 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.556433 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.556443 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.557055 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:03.557946 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.557975 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.559293 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.559297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.560986 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.560998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.561788 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.561793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.562041 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.562842 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:03.562869 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:03.564813 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:03.566695 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.566701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.566719 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.567021 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.567329 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.567333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.567335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.567484 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.567918 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:03.567927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:03.568193 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.568458 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.568814 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":118,"threads_started":1}
I20260928 23:58:03.570945 4623 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:58:03.571276 4623 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:03.571808 4623 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
I20260928 23:58:03.573021 4295 ts_manager.cc:194] Registered new tserver with Master: 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183)
I20260928 23:58:03.573805 4295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:37111
W20260928 23:58:03.674431 4626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:03.674732 4626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:03.674778 4626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:03.679040 4626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:03.679108 4626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:03.679183 4626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:03.684799 4626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=34711
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:03.685896 4626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:03.686889 4626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:03.697762 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.697795 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:03.697821 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:03.697885 4626 server_base.cc:1034] running on GCE node
I20260928 23:58:03.698509 4626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:03.699282 4626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:03.700469 4626 hybrid_clock.cc:648] HybridClock initialized: now 1790639883700442 us; error 51 us; skew 500 ppm
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:03.703578 4626 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:58:03.704624 4626 webserver.cc:460] Webserver started at http://127.20.88.3:34711/ using document root <none> and password file <none>
I20260928 23:58:03.705262 4626 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:03.705344 4626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:03.709847 4626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:58:03.712144 4641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.713549 4626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:03.713703 4626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "aea2317bbecb414297abec899caf8d40"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "704ae11508ad92fa56ca13e0945555ea"
server_key_iv: "949f98751d7e04c67841b7200e297158"
server_key_version: "encryptionkey@0"
I20260928 23:58:03.714336 4626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:03.726728 4626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:03.730926 4626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:03.731249 4626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:03.731974 4626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:03.732993 4626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:03.733075 4626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.733153 4626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:03.733201 4626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:03.743829 4626 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:35045
I20260928 23:58:03.743849 4754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:35045 every 8 connection(s)
I20260928 23:58:03.745128 4626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:03.750491 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4626
Sep 28 23:58:03 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639883, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:03.760390 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.751257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.20.88.62:35997)
0928 23:58:03.751530 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.751535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.751652 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:58:03.751813 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.751816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.751872 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.751972 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.752795 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.753334 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.754191 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.754426 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.754541 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.756218 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.756222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.756224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.756304 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.758429 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.758976 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.758980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.758982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.759088 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.759381 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.759385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.759386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.759649 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.759899 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.760020 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.760162 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260928 23:58:03.761439 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.747555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:57193 (local address 127.20.88.62:35997)
0928 23:58:03.747758 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.747764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.748699 (+ 935us) server_negotiation.cc:407] Connection header received
0928 23:58:03.750096 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.750099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.750190 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.750299 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.751928 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.752546 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.753800 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.754025 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.754164 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.757064 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.757073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.757075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.757110 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.758833 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.759586 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.759589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.759590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.759663 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.760099 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.760106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.760111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.760378 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.760482 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.761180 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.761326 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 23:58:03.762629 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.747936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:57193)
0928 23:58:03.748493 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.748540 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.749872 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.750474 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.750482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.751059 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:03.751750 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.751766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.752695 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.752703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.753658 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.753669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.754230 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.754235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.754423 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.755136 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:03.755161 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:03.756882 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:03.758983 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.758991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.759012 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.759482 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.759786 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:03.759789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:03.759791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:03.759988 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:03.760474 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:03.760485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:03.760893 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.761111 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.761515 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":123,"threads_started":1}
I20260928 23:58:03.764065 4755 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35997
I20260928 23:58:03.764438 4755 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:03.764966 4755 heartbeater.cc:511] Master 127.20.88.62:35997 requested a full tablet report, sending...
I20260928 23:58:03.766239 4295 ts_manager.cc:194] Registered new tserver with Master: aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
I20260928 23:58:03.767004 4295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:57193
I20260928 23:58:03.774387 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:03.778745 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.775047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.20.88.62:35997)
0928 23:58:03.775198 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.775202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.775461 (+ 259us) server_negotiation.cc:407] Connection header received
0928 23:58:03.775604 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.775606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.775737 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.775827 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:03.776035 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.776656 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.777531 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.777773 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.777862 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.778217 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:03.778359 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 23:58:03.778572 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58172: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:58:03.778833 4487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58172: token expired
W20260928 23:58:03.779162 4226 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.88.62:35997)
I20260928 23:58:03.786473 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.779551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.20.88.62:35997)
0928 23:58:03.779681 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.779685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.779697 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:58:03.779809 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.779811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.779850 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.779914 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:03.780573 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.781063 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.781853 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.782058 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.782138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.783490 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:03.783492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:03.783494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:03.783514 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:03.784923 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.785335 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.785338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.785339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.785422 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:03.785728 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:03.785731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:03.785733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:03.785917 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:03.786039 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.786254 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.786352 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:58:03.789225 4295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:03.792901 4295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:03.802744 4419 tablet_service.cc:1467] Processing CreateTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c (DEFAULT_TABLE table=test-table [id=be7e2bb897ae4832a8bd2b96e7536b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:03.803509 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3678f084f4d54108a4dbf5a35b8d3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:03.806546 4377 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap starting.
I20260928 23:58:03.807474 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.801332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:35045 (local address 127.0.0.1:41976)
0928 23:58:03.801643 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.801661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.801759 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.802960 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.802965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.802985 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:03.803080 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.803086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.804716 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.804719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.805817 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.805826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.806932 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.806936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.807057 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.807118 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.807216 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.807308 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260928 23:58:03.807482 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.801892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.20.88.3:35045)
0928 23:58:03.802324 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.802336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.802361 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:58:03.802437 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.802442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.802635 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.803354 (+ 719us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:03.803537 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.804593 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.805945 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.806794 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.806916 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.807074 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.807258 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.807352 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":191,"threads_started":1}
I20260928 23:58:03.807569 4377 tablet_bootstrap.cc:654] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:03.808315 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.800995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38183 (local address 127.0.0.1:37916)
0928 23:58:03.801147 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.801165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.801260 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.802689 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.802692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.802720 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:03.802860 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.802868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.804379 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.804382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.805726 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.805739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.807776 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.807781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.807893 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.807924 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.808020 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.808133 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:58:03.808634 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.801500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.20.88.2:38183)
0928 23:58:03.801946 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.801963 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.802067 (+ 104us) server_negotiation.cc:407] Connection header received
0928 23:58:03.802204 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.802210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.802489 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.803083 (+ 594us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:03.803299 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.804207 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.805888 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.806794 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.806916 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.807074 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.808072 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.808184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":203,"threads_started":1}
I20260928 23:58:03.810535 4377 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: No bootstrap required, opened a new log
I20260928 23:58:03.810654 4377 ts_tablet_manager.cc:1397] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:58:03.811153 4377 raft_consensus.cc:348] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.811296 4377 raft_consensus.cc:374] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:03.811337 4377 raft_consensus.cc:729] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f97c6ebe1e4f6997b9001536f6e0ea, State: Initialized, Role: FOLLOWER
I20260928 23:58:03.810959 4689 tablet_service.cc:1467] Processing CreateTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c (DEFAULT_TABLE table=test-table [id=be7e2bb897ae4832a8bd2b96e7536b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:03.811471 4377 consensus_queue.cc:260] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.811776 4377 ts_tablet_manager.cc:1428] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:58:03.811743 4557 tablet_service.cc:1467] Processing CreateTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c (DEFAULT_TABLE table=test-table [id=be7e2bb897ae4832a8bd2b96e7536b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:03.812337 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3678f084f4d54108a4dbf5a35b8d3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:03.812784 4557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3678f084f4d54108a4dbf5a35b8d3c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:03.819998 4768 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Bootstrap starting.
I20260928 23:58:03.820353 4769 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Bootstrap starting.
I20260928 23:58:03.822263 4769 tablet_bootstrap.cc:654] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:03.822696 4768 tablet_bootstrap.cc:654] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:03.823422 4768 log.cc:824] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:03.823423 4769 log.cc:824] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:03.825600 4769 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: No bootstrap required, opened a new log
I20260928 23:58:03.825599 4768 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: No bootstrap required, opened a new log
I20260928 23:58:03.825824 4768 ts_tablet_manager.cc:1397] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:58:03.825825 4769 ts_tablet_manager.cc:1397] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:03.828831 4769 raft_consensus.cc:348] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.828831 4768 raft_consensus.cc:348] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.829074 4769 raft_consensus.cc:374] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:03.829101 4768 raft_consensus.cc:374] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:03.829150 4769 raft_consensus.cc:729] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea2317bbecb414297abec899caf8d40, State: Initialized, Role: FOLLOWER
I20260928 23:58:03.829165 4768 raft_consensus.cc:729] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ec400d8f0e41cbb7cb9b23e356d9e7, State: Initialized, Role: FOLLOWER
I20260928 23:58:03.829576 4768 consensus_queue.cc:260] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.829603 4769 consensus_queue.cc:260] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.830227 4755 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
I20260928 23:58:03.830232 4623 heartbeater.cc:503] Master 127.20.88.62:35997 was elected leader, sending a full tablet report...
I20260928 23:58:03.830606 4768 ts_tablet_manager.cc:1428] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:58:03.831562 4769 ts_tablet_manager.cc:1428] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 23:58:03.835847 4486 tablet.cc:2369] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:03.856913 4772 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:03.857203 4772 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.858464 4772 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045), 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:58:03.863240 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.858804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:56019 (local address 127.20.88.3:35045)
0928 23:58:03.859107 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.859112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.859129 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:58:03.859188 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.859191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.859248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.859335 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:03.859636 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.860444 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862078 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.862608 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862739 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.862803 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.863044 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.863120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260928 23:58:03.863238 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.859180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:34601 (local address 127.20.88.1:44005)
0928 23:58:03.859317 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.859324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.859458 (+ 134us) server_negotiation.cc:407] Connection header received
0928 23:58:03.859539 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.859541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.859626 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.859726 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:03.859966 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.860686 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862117 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.862590 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862737 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.862931 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.863022 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.863096 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:58:03.863243 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.858730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:35045 (local address 127.20.88.2:56019)
0928 23:58:03.858853 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.858870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.858975 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.859349 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.859352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.859373 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:03.859519 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.859527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.860590 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.860617 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.861921 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.861934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.862748 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.862916 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.862934 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.863025 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.863110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":37}
I20260928 23:58:03.863255 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.859031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.20.88.2:34601)
0928 23:58:03.859356 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.859368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:03.859457 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:03.859742 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:03.859744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:03.859761 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:03.859856 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.859860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.860795 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.860798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.861966 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:03.861976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.862738 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.862742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:03.862891 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.862920 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:58:03.863005 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:58:03.863085 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":4,"thread_start_us":146,"threads_started":1}
I20260928 23:58:03.864010 4709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3678f084f4d54108a4dbf5a35b8d3c1c" candidate_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea2317bbecb414297abec899caf8d40" is_pre_election: true
I20260928 23:58:03.864001 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3678f084f4d54108a4dbf5a35b8d3c1c" candidate_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" is_pre_election: true
I20260928 23:58:03.864362 4709 raft_consensus.cc:2454] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ec400d8f0e41cbb7cb9b23e356d9e7 in term 0.
I20260928 23:58:03.864362 4439 raft_consensus.cc:2454] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ec400d8f0e41cbb7cb9b23e356d9e7 in term 0.
I20260928 23:58:03.864868 4512 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, 67f97c6ebe1e4f6997b9001536f6e0ea; no voters:
I20260928 23:58:03.865207 4772 raft_consensus.cc:2781] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:03.865326 4772 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:03.865388 4772 raft_consensus.cc:3037] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:03.866571 4772 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.866938 4772 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [CANDIDATE]: Term 1 election: Requested vote from peers aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045), 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005)
I20260928 23:58:03.867431 4709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3678f084f4d54108a4dbf5a35b8d3c1c" candidate_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea2317bbecb414297abec899caf8d40"
I20260928 23:58:03.867441 4439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3678f084f4d54108a4dbf5a35b8d3c1c" candidate_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
I20260928 23:58:03.867586 4439 raft_consensus.cc:3037] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:03.867584 4709 raft_consensus.cc:3037] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:03.868891 4709 raft_consensus.cc:2454] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ec400d8f0e41cbb7cb9b23e356d9e7 in term 1.
I20260928 23:58:03.868894 4439 raft_consensus.cc:2454] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ec400d8f0e41cbb7cb9b23e356d9e7 in term 1.
I20260928 23:58:03.869200 4512 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, 67f97c6ebe1e4f6997b9001536f6e0ea; no voters:
I20260928 23:58:03.869382 4772 raft_consensus.cc:2781] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:03.869606 4772 raft_consensus.cc:686] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 LEADER]: Becoming Leader. State: Replica: 34ec400d8f0e41cbb7cb9b23e356d9e7, State: Running, Role: LEADER
I20260928 23:58:03.869975 4772 consensus_queue.cc:237] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:03.873513 4297 catalog_manager.cc:5495] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } health_report { overall_health: UNKNOWN } } }
I20260928 23:58:03.890434 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.886681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.20.88.2:38183)
0928 23:58:03.886834 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:03.886838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:03.886856 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:58:03.886962 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:03.886964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:03.887020 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:03.887152 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:03.887430 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.887931 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.888875 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:03.889104 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:03.889217 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:03.889550 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:03.890001 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:03.890150 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:03.890186 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:58:03.890249 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:58:03.897722 4709 raft_consensus.cc:1261] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Refusing update from remote peer 34ec400d8f0e41cbb7cb9b23e356d9e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:03.897722 4439 raft_consensus.cc:1261] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Refusing update from remote peer 34ec400d8f0e41cbb7cb9b23e356d9e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:03.898542 4775 consensus_queue.cc:1035] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:03.898778 4772 consensus_queue.cc:1035] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:03.914014 4297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:03.914218 4297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:03.916846 4297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Sending DeleteTablet for 3 replicas of tablet 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:03.917788 4557 tablet_service.cc:1514] Processing DeleteTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:03 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:37916
I20260928 23:58:03.917794 4689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:03 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:41976
I20260928 23:58:03.918298 4419 tablet_service.cc:1514] Processing DeleteTablet for tablet 3678f084f4d54108a4dbf5a35b8d3c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:03 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:37734
I20260928 23:58:03.918359 4787 tablet_replica.cc:330] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: stopping tablet replica
I20260928 23:58:03.918381 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4335
I20260928 23:58:03.918874 4788 tablet_replica.cc:330] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: stopping tablet replica
I20260928 23:58:03.919198 4788 raft_consensus.cc:2229] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:03.919598 4788 raft_consensus.cc:2258] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:03.919950 4787 raft_consensus.cc:2229] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:03.920382 4787 raft_consensus.cc:2258] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:03.921481 4788 ts_tablet_manager.cc:1905] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:03.921933 4787 ts_tablet_manager.cc:1905] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:03.924932 4787 ts_tablet_manager.cc:1918] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:03.925158 4787 log.cc:1196] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:03.925370 4788 ts_tablet_manager.cc:1918] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:03.925597 4788 log.cc:1196] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:03.925643 4787 ts_tablet_manager.cc:1939] T 3678f084f4d54108a4dbf5a35b8d3c1c P aea2317bbecb414297abec899caf8d40: Deleting consensus metadata
I20260928 23:58:03.926041 4788 ts_tablet_manager.cc:1939] T 3678f084f4d54108a4dbf5a35b8d3c1c P 34ec400d8f0e41cbb7cb9b23e356d9e7: Deleting consensus metadata
I20260928 23:58:03.927181 4283 catalog_manager.cc:4841] TS aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): tablet 3678f084f4d54108a4dbf5a35b8d3c1c (table test-table [id=be7e2bb897ae4832a8bd2b96e7536b1f]) successfully deleted
I20260928 23:58:03.927363 4281 catalog_manager.cc:4841] TS 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): tablet 3678f084f4d54108a4dbf5a35b8d3c1c (table test-table [id=be7e2bb897ae4832a8bd2b96e7536b1f]) successfully deleted
W20260928 23:58:03.927736 4283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:44005 (error 108)
I20260928 23:58:03.928054 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4489
I20260928 23:58:03.930299 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:03.929978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:44005 (local address 127.0.0.1:37748)
0928 23:58:03.930128 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:03.930207 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 23:58:03.930500 4283 catalog_manager.cc:4584] TS 67f97c6ebe1e4f6997b9001536f6e0ea (127.20.88.1:44005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3678f084f4d54108a4dbf5a35b8d3c1c: Network error: Client connection negotiation failed: client connection to 127.20.88.1:44005: connect: Connection refused (error 111)
I20260928 23:58:03.933972 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4626
I20260928 23:58:03.939641 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4264
I20260928 23:58:23.946805 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--webserver_interface=127.20.88.62
--webserver_port=41347
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:24.065301 4790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:24.065624 4790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:24.065670 4790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:24.070016 4790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:58:24.070122 4790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:58:24.070138 4790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:58:24.070154 4790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:24.070174 4790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:24.070194 4790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:24.070209 4790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:24.070225 4790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:24.076067 4790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:24.077250 4790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:24.078282 4790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:24.087745 4796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.087760 4795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.087756 4798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:24.088011 4790 server_base.cc:1034] running on GCE node
I20260928 23:58:24.088778 4790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:24.089962 4790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:24.091140 4790 hybrid_clock.cc:648] HybridClock initialized: now 1790639904091123 us; error 31 us; skew 500 ppm
Sep 28 23:58:24 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:24.098315 4790 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:58:24.099454 4790 webserver.cc:460] Webserver started at http://127.20.88.62:41347/ using document root <none> and password file <none>
I20260928 23:58:24.100080 4790 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:24.100152 4790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:24.104579 4790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:24.107002 4805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.108659 4790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:24.108829 4790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "f1561f1870fb462d85456a14501198dd"
format_stamp: "Formatted at 2026-09-28 23:57:41 on dist-test-slave-wvpw"
server_key: "04082f165927937a731fb3c040b0b87a"
server_key_iv: "b1c80c52256e325820511eb79728b973"
server_key_version: "encryptionkey@0"
I20260928 23:58:24.109478 4790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:24.122201 4790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:24.126843 4790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:24.127074 4790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:24.135991 4790 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35997
I20260928 23:58:24.136375 4857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35997 every 8 connection(s)
I20260928 23:58:24.137807 4790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:24.142702 4858 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:58:24.144567 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4790
I20260928 23:58:24.144783 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap starting.
I20260928 23:58:24.145094 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:44005
--local_ip_for_outbound_sockets=127.20.88.1
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=38195
--webserver_interface=127.20.88.1
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:58:24.163750 4858 log.cc:824] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:24.175558 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:58:24.176014 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Bootstrap complete.
I20260928 23:58:24.180356 4858 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:24.180836 4858 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1561f1870fb462d85456a14501198dd, State: Initialized, Role: FOLLOWER
I20260928 23:58:24.181351 4858 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:24.181473 4858 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:24.181569 4858 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:24.181689 4858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:58:24.183748 4858 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:24.184119 4858 leader_election.cc:302] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1561f1870fb462d85456a14501198dd; no voters:
I20260928 23:58:24.184512 4858 leader_election.cc:288] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:58:24.184607 4862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:58:24.184798 4862 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [term 3 LEADER]: Becoming Leader. State: Replica: f1561f1870fb462d85456a14501198dd, State: Running, Role: LEADER
I20260928 23:58:24.185107 4862 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } }
I20260928 23:58:24.185849 4858 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:24.186126 4862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1561f1870fb462d85456a14501198dd. Latest consensus state: current_term: 3 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:58:24.186250 4862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:24.186651 4863 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f1561f1870fb462d85456a14501198dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1561f1870fb462d85456a14501198dd" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35997 } } }
I20260928 23:58:24.186822 4863 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:24.188066 4866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:24.192834 4866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:24.193477 4866 catalog_manager.cc:1259] Loaded cluster ID: 7da02be5318c4eb38e2f9c03810c6485
I20260928 23:58:24.193564 4866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:24.195411 4866 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:24.196151 4866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Loaded TSK: 0
I20260928 23:58:24.196272 4866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1561f1870fb462d85456a14501198dd: Loaded TSK: 1
F20260928 23:58:24.197316 4866 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbdff4ac9ad google::LogMessage::Fail() at ??:0
@ 0x7fbdff4af88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbdff4ac4de google::LogMessage::Flush() at ??:0
@ 0x7fbdff4ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe0217018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbe02168f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbe09e117bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbe09deb99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbe09dec178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbe09e2ddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbe09e4c82b std::function<>::operator()() at ??:0
@ 0x7fbe09debb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbe09decf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe09de9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbe09e2d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe016fe3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe016fec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbe017003f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe016ef512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe02d336db start_thread at ??:0
@ 0x7fbdfdfaa61f clone at ??:0
W20260928 23:58:24.308665 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:24.309118 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:24.309192 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:24.316026 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:24.316148 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:24.316273 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:24.325125 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:44005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=38195
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:24.326659 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:24.328073 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:24.340581 4906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.340550 4905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.340550 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:24.341195 4860 server_base.cc:1034] running on GCE node
I20260928 23:58:24.341724 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:24.342587 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:24.343830 4860 hybrid_clock.cc:648] HybridClock initialized: now 1790639904343810 us; error 39 us; skew 500 ppm
Sep 28 23:58:24 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:24.346840 4860 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:58:24.347849 4860 webserver.cc:460] Webserver started at http://127.20.88.1:38195/ using document root <none> and password file <none>
I20260928 23:58:24.348562 4860 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:24.348649 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:24.352969 4860 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:24.355618 4915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.357296 4860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:58:24.357502 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "67f97c6ebe1e4f6997b9001536f6e0ea"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "2de13984a4fb0c2cc8cfd72330321692"
server_key_iv: "4e7361c2be2c8de94d18aa284b33cda9"
server_key_version: "encryptionkey@0"
I20260928 23:58:24.358129 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:24.376997 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:24.381101 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:24.381458 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:24.382169 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:24.383412 4922 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:58:24.389130 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:58:24.389252 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 23:58:24.389345 4860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:58:24.391173 4860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:58:24.391261 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:58:24.391598 4922 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap starting.
I20260928 23:58:24.403357 4860 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:44005
I20260928 23:58:24.403443 5029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:44005 every 8 connection(s)
I20260928 23:58:24.404562 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:24.413628 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 4860
I20260928 23:58:24.414127 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:38183
--local_ip_for_outbound_sockets=127.20.88.2
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=44329
--webserver_interface=127.20.88.2
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:58:24.415985 4922 log.cc:824] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:24.426321 4922 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:58:24.426891 4922 tablet_bootstrap.cc:492] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Bootstrap complete.
I20260928 23:58:24.428414 4922 ts_tablet_manager.cc:1397] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.011s
I20260928 23:58:24.432132 4922 raft_consensus.cc:348] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:24.432787 4922 raft_consensus.cc:729] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f97c6ebe1e4f6997b9001536f6e0ea, State: Initialized, Role: FOLLOWER
I20260928 23:58:24.433403 4922 consensus_queue.cc:260] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:24.434592 4922 ts_tablet_manager.cc:1428] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 23:58:24.460178 5032 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:58:24.406705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:42785)
0928 23:58:24.407248 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.407309 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:24.408192 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:24.458949 (+ 50757us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":176,"threads_started":1}
W20260928 23:58:24.460930 5030 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:35997 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:35997: Client connection negotiation failed: client connection to 127.20.88.62:35997: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:58:24.461971 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.461618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:54623)
0928 23:58:24.461805 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.461890 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:58:24.462811 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.462530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:55233)
0928 23:58:24.462686 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.462740 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 23:58:24.463022 5030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:58:24.537948 5033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:24.538298 5033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:24.538388 5033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:24.542888 5033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:24.543008 5033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:24.543110 5033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:24.548988 5033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:38183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=44329
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:24.550246 5033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:24.551394 5033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:24.562403 5043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.562420 5041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.562403 5040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:24.562638 5033 server_base.cc:1034] running on GCE node
I20260928 23:58:24.563272 5033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:24.564142 5033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:24.565459 5033 hybrid_clock.cc:648] HybridClock initialized: now 1790639904565440 us; error 32 us; skew 500 ppm
Sep 28 23:58:24 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:24.568533 5033 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:58:24.569442 5033 webserver.cc:460] Webserver started at http://127.20.88.2:44329/ using document root <none> and password file <none>
I20260928 23:58:24.569993 5033 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:24.570063 5033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:24.574265 5033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:58:24.576596 5050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.577976 5033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:58:24.578176 5033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "d588be444113528a242fdebc328b793f"
server_key_iv: "33d9c30fcf3b10e6d803e7934aa4b816"
server_key_version: "encryptionkey@0"
I20260928 23:58:24.578816 5033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:24.617406 5033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:24.621018 5033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:24.621239 5033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:24.621862 5033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:24.622735 5033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:24.622805 5033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.622881 5033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:24.622915 5033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.632464 5033 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:38183
I20260928 23:58:24.632536 5163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:38183 every 8 connection(s)
I20260928 23:58:24.633603 5033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:24.637917 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.635982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:53235)
0928 23:58:24.636580 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.636702 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":158,"threads_started":1}
W20260928 23:58:24.638420 5164 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:35997 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:35997: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
I20260928 23:58:24.639086 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.638853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:54517)
0928 23:58:24.638969 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.639025 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:58:24.639700 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.639513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:39955)
0928 23:58:24.639612 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.639654 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:58:24.639879 5164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:58:24.642617 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5033
I20260928 23:58:24.643046 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:35045
--local_ip_for_outbound_sockets=127.20.88.3
--tserver_master_addrs=127.20.88.62:35997
--webserver_port=34711
--webserver_interface=127.20.88.3
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:24.759807 5167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:24.760083 5167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:24.760128 5167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:24.764417 5167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:24.764493 5167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:24.764564 5167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:24.770259 5167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:35045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=34711
--tserver_master_addrs=127.20.88.62:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:24.771409 5167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:24.772593 5167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:24.783128 5173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.783128 5172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:24.783138 5175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:24.783638 5167 server_base.cc:1034] running on GCE node
I20260928 23:58:24.784181 5167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:24.784982 5167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:24.786180 5167 hybrid_clock.cc:648] HybridClock initialized: now 1790639904786163 us; error 32 us; skew 500 ppm
Sep 28 23:58:24 dist-test-slave-wvpw krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:24.789290 5167 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:58:24.790376 5167 webserver.cc:460] Webserver started at http://127.20.88.3:34711/ using document root <none> and password file <none>
I20260928 23:58:24.791013 5167 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:24.791115 5167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:24.795671 5167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:24.798225 5182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.799693 5167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:24.799832 5167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "aea2317bbecb414297abec899caf8d40"
format_stamp: "Formatted at 2026-09-28 23:57:42 on dist-test-slave-wvpw"
server_key: "704ae11508ad92fa56ca13e0945555ea"
server_key_iv: "949f98751d7e04c67841b7200e297158"
server_key_version: "encryptionkey@0"
I20260928 23:58:24.800521 5167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:24.813688 5167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:24.817083 5167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:24.817255 5167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:24.817814 5167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:24.818653 5167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:24.818703 5167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.818746 5167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:24.818763 5167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:24.829348 5167 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:35045
I20260928 23:58:24.829366 5295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:35045 every 8 connection(s)
I20260928 23:58:24.830427 5167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:24.830811 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5167
I20260928 23:58:24.834697 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.832833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:38231)
0928 23:58:24.833304 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.833419 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
W20260928 23:58:24.835453 5296 heartbeater.cc:644] Failed to heartbeat to 127.20.88.62:35997 (0 consecutive failures): Network error: Failed to ping master at 127.20.88.62:35997: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
I20260928 23:58:24.836208 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.835962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:39169)
0928 23:58:24.836089 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.836144 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:58:24.836885 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:24.836680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:56847)
0928 23:58:24.836781 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:24.836825 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:58:24.837101 5296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:58:25.464329 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.463718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:57037)
0928 23:58:25.464075 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:25.464140 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
I20260928 23:58:25.641094 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.640598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:54173)
0928 23:58:25.640914 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:25.640982 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260928 23:58:25.681370 5302 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:25.681573 5302 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:25.682833 5302 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
Sep 28 23:58:25 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.3@KRBTEST.COM
Sep 28 23:58:25 dist-test-slave-wvpw krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.2@KRBTEST.COM
I20260928 23:58:25.702353 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.683212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:57835 (local address 127.20.88.2:38183)
0928 23:58:25.683616 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:25.683645 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:25.683691 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:58:25.684653 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:25.684664 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:25.685151 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:25.685503 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:25.687091 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.688085 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.689461 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.689980 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.690156 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:25.697220 (+ 7064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:25.697228 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:25.697245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:25.697288 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:25.699310 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:25.699929 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:25.699937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:25.699944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:25.700120 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:25.700604 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:25.700608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:25.700610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:25.701176 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:25.701286 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:25.701886 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 23:58:25.702168 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260928 23:58:25.702339 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.683199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:35045 (local address 127.20.88.1:35373)
0928 23:58:25.683614 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:25.683629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:25.683737 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:25.686000 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:25.686003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:25.686266 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:25.686898 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:25.686915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.688286 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.688289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:25.689315 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:25.689327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.690141 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.690144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:25.690304 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:25.694625 (+ 4321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:25.694658 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:25.696680 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:25.699935 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:25.699937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:25.699939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:25.700172 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:25.700610 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:25.700613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:25.700616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:25.700726 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:25.701366 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:25.701368 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:25.701563 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 23:58:25.701753 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 23:58:25.702140 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":166,"spinlock_wait_cycles":12928,"thread_start_us":70,"threads_started":1}
I20260928 23:58:25.702399 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.683370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:35373 (local address 127.20.88.3:35045)
0928 23:58:25.683793 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:25.683804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:25.683838 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:58:25.684653 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:25.684664 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:25.685151 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:25.685503 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:25.687117 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.688089 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.689519 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.689978 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.690187 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:25.697056 (+ 6869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:25.697065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:25.697086 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:25.697131 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:25.699304 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:25.700310 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:25.700316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:25.700322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:25.700478 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:25.700832 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:25.700834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:25.700835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:25.701225 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:25.701381 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:25.701892 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 23:58:25.702204 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":108,"threads_started":1}
I20260928 23:58:25.702386 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.683052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:38183 (local address 127.20.88.1:57835)
0928 23:58:25.683179 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:25.683199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:25.683315 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:25.685592 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:25.685598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:25.686238 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:25.686898 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:25.686915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.688278 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.688283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:25.689315 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:25.689327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:25.690122 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:25.690126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:25.690301 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:25.694625 (+ 4324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:25.694653 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:25.696986 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:25.699474 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:25.699481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:25.699500 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:25.699790 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:25.700291 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:25.700295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:25.700297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:25.700481 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:25.701303 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:25.701313 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:25.701571 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 23:58:25.701754 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 23:58:25.702228 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":301,"spinlock_wait_cycles":29184}
W20260928 23:58:25.705295 4918 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
W20260928 23:58:25.705544 4916 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:25.705670 4916 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea; no voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40
I20260928 23:58:25.705968 5302 raft_consensus.cc:2726] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:58:25.838271 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:25.837802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:51933)
0928 23:58:25.838106 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:25.838170 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
I20260928 23:58:26.465710 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:26.465264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:33159)
0928 23:58:26.465558 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:26.465614 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260928 23:58:26.642453 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:26.641988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:51675)
0928 23:58:26.642265 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:26.642336 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260928 23:58:26.839792 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:26.839286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:42043)
0928 23:58:26.839603 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:26.839672 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260928 23:58:27.209859 5310 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:27.210004 5310 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:27.210566 5310 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
W20260928 23:58:27.211642 4918 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
W20260928 23:58:27.211793 4916 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:27.211889 4916 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea; no voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40
I20260928 23:58:27.212047 5310 raft_consensus.cc:2726] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:58:27.467319 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:27.466694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:52151)
0928 23:58:27.467081 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:27.467164 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":89,"threads_started":1}
I20260928 23:58:27.643882 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:27.643405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:33319)
0928 23:58:27.643710 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:27.643779 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260928 23:58:27.841248 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:27.840748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:54929)
0928 23:58:27.841072 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:27.841147 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260928 23:58:28.468832 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:28.468384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:33199)
0928 23:58:28.468668 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:28.468732 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260928 23:58:28.645321 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:28.644869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:42867)
0928 23:58:28.645153 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:28.645221 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260928 23:58:28.842687 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:28.842249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:35639)
0928 23:58:28.842517 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:28.842584 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260928 23:58:29.470459 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:29.469925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:55019)
0928 23:58:29.470257 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:29.470353 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
I20260928 23:58:29.646680 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:29.646223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:44519)
0928 23:58:29.646514 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:29.646580 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260928 23:58:29.843962 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:29.843516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:47927)
0928 23:58:29.843795 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:29.843860 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260928 23:58:29.875381 5320 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:29.875578 5320 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:29.876127 5320 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
W20260928 23:58:29.877023 4916 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
W20260928 23:58:29.877187 4918 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:29.877293 4918 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea; no voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40
I20260928 23:58:29.877550 5320 raft_consensus.cc:2726] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:58:30.471956 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:30.471461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:38161)
0928 23:58:30.471789 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:30.471856 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260928 23:58:30.648288 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:30.647729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:46709)
0928 23:58:30.648061 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:30.648124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":142,"threads_started":1}
I20260928 23:58:30.845446 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:30.844873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:58387)
0928 23:58:30.845236 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:30.845306 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":140,"threads_started":1}
I20260928 23:58:31.473476 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:31.472996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:48361)
0928 23:58:31.473297 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:31.473361 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260928 23:58:31.649837 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:31.649290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:55065)
0928 23:58:31.649681 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:31.649739 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":98,"threads_started":1}
I20260928 23:58:31.846802 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:31.846301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:42091)
0928 23:58:31.846633 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:31.846700 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20260928 23:58:32.474854 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:32.474365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:35787)
0928 23:58:32.474694 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:32.474755 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260928 23:58:32.651108 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:32.650692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:41639)
0928 23:58:32.650938 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:32.650988 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260928 23:58:32.848380 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:32.847838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:46481)
0928 23:58:32.848163 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:32.848254 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260928 23:58:32.938987 5330 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:32.939208 5330 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:32.939741 5330 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
W20260928 23:58:32.940629 4916 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
W20260928 23:58:32.940748 4918 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:32.940869 4918 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea; no voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40
I20260928 23:58:32.941030 5330 raft_consensus.cc:2726] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:58:33.476491 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:33.475933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:35545)
0928 23:58:33.476312 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:33.476376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260928 23:58:33.652379 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:33.651905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:40647)
0928 23:58:33.652161 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:33.652215 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260928 23:58:33.849812 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:33.849336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:41999)
0928 23:58:33.849622 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:33.849694 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260928 23:58:34.477909 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:34.477418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:46837)
0928 23:58:34.477738 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:34.477805 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260928 23:58:34.653682 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:34.653211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:33481)
0928 23:58:34.653528 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:34.653584 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":86,"threads_started":1}
I20260928 23:58:34.851274 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:34.850783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:56929)
0928 23:58:34.851095 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:34.851172 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260928 23:58:35.479303 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:35.478877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:47949)
0928 23:58:35.479138 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:35.479200 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260928 23:58:35.655220 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:35.654661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:44295)
0928 23:58:35.655013 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:35.655091 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260928 23:58:35.852954 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:35.852320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:58829)
0928 23:58:35.852700 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:35.852799 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260928 23:58:36.480902 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:36.480426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:50655)
0928 23:58:36.480728 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:36.480795 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260928 23:58:36.656512 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:36.656060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:44833)
0928 23:58:36.656359 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:36.656415 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":83,"threads_started":1}
I20260928 23:58:36.854451 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:36.853946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:59481)
0928 23:58:36.854275 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:36.854350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260928 23:58:37.482349 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:37.481874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:42731)
0928 23:58:37.482190 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:37.482251 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260928 23:58:37.657684 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:37.657239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:57439)
0928 23:58:37.657522 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:37.657583 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260928 23:58:37.855645 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:37.855244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:57819)
0928 23:58:37.855489 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:37.855547 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260928 23:58:37.890709 5346 raft_consensus.cc:482] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:37.890869 5346 raft_consensus.cc:504] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34ec400d8f0e41cbb7cb9b23e356d9e7" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 38183 } } peers { permanent_uuid: "aea2317bbecb414297abec899caf8d40" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 35045 } } peers { permanent_uuid: "67f97c6ebe1e4f6997b9001536f6e0ea" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 44005 } }
I20260928 23:58:37.891356 5346 leader_election.cc:288] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183), aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045)
W20260928 23:58:37.892179 4918 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aea2317bbecb414297abec899caf8d40 (127.20.88.3:35045): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
W20260928 23:58:37.892345 4916 leader_election.cc:341] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ec400d8f0e41cbb7cb9b23e356d9e7 (127.20.88.2:38183): Not found: Tablet not found: 3678f084f4d54108a4dbf5a35b8d3c1c
I20260928 23:58:37.892417 4916 leader_election.cc:302] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67f97c6ebe1e4f6997b9001536f6e0ea; no voters: 34ec400d8f0e41cbb7cb9b23e356d9e7, aea2317bbecb414297abec899caf8d40
I20260928 23:58:37.892557 5346 raft_consensus.cc:2726] T 3678f084f4d54108a4dbf5a35b8d3c1c P 67f97c6ebe1e4f6997b9001536f6e0ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:58:38.483767 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:38.483276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:53957)
0928 23:58:38.483597 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:38.483665 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":73,"threads_started":1}
I20260928 23:58:38.659178 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:38.658669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:58351)
0928 23:58:38.659001 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:38.659071 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260928 23:58:38.857000 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:38.856534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.3:60531)
0928 23:58:38.856823 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:38.856890 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260928 23:58:39.485430 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:39.484952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.1:55695)
0928 23:58:39.485257 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:39.485322 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":76,"threads_started":1}
I20260928 23:58:39.660861 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:39.660366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35997 (local address 127.20.88.2:39259)
0928 23:58:39.660651 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:39.660712 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.62:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260928 23:58:39.837785 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 4860
I20260928 23:58:39.844041 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5033
I20260928 23:58:39.849481 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5167
2026-09-28T23:58:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59391 ms)
[----------] 30 tests from SecurityITest (250739 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:39 dist-test-slave-wvpw krb5kdc[5354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:39 dist-test-slave-wvpw krb5kdc[5354](info): set up 2 sockets
Sep 28 23:58:39 dist-test-slave-wvpw krb5kdc[5354](info): commencing operation
krb5kdc: starting...
W20260928 23:58:40.908610 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:40 dist-test-slave-wvpw krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:58:41.044026 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41077
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:41.162519 5367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:41.162833 5367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:41.162878 5367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:41.167083 5367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:41.167171 5367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:41.167191 5367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:41.167214 5367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:41.167230 5367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:41.173548 5367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:41077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:41.174713 5367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:41.175765 5367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:41.185237 5373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.185228 5375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.185297 5367 server_base.cc:1034] running on GCE node
W20260928 23:58:41.185230 5372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.186110 5367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:41.187340 5367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:41.188536 5367 hybrid_clock.cc:648] HybridClock initialized: now 1790639921188502 us; error 48 us; skew 500 ppm
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:41.191581 5367 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:58:41.192684 5367 webserver.cc:460] Webserver started at http://127.20.88.62:41119/ using document root <none> and password file <none>
I20260928 23:58:41.193321 5367 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:41.193419 5367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:41.193717 5367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:41.195993 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "74aa895ebf230397f177f5037e9dc3bd"
server_key_iv: "c1487e8d403eab320fb7e58d1a9362fe"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.196666 5367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "74aa895ebf230397f177f5037e9dc3bd"
server_key_iv: "c1487e8d403eab320fb7e58d1a9362fe"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.200781 5367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:41.203312 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.204689 5367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:58:41.204841 5367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "74aa895ebf230397f177f5037e9dc3bd"
server_key_iv: "c1487e8d403eab320fb7e58d1a9362fe"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.204972 5367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:41.214603 5367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:41.218909 5367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:41.219126 5367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:41.226173 5367 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:41077
I20260928 23:58:41.226214 5434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:41077 every 8 connection(s)
I20260928 23:58:41.227160 5367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:41.230088 5435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:41.230947 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5367
I20260928 23:58:41.231088 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:41.231384 20832 external_mini_cluster.cc:1428] Setting key 5e80a374950929bddb5ddf2954b7e997
I20260928 23:58:41.237126 5435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: Bootstrap starting.
I20260928 23:58:41.239950 5435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:41.240855 5435 log.cc:824] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:41.243446 5435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: No bootstrap required, opened a new log
I20260928 23:58:41.247489 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:41.232866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.20.88.62:41077)
0928 23:58:41.233300 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:41.233320 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:41.233366 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:58:41.234237 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:41.234261 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:41.234717 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:41.235113 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:41.235835 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.237349 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.238255 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.238965 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.239136 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:41.242096 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:41.242102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:41.242117 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:41.242146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:41.244133 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.244750 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.244793 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.244798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.244925 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.245280 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.245283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.245284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.245726 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:41.245932 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:41.246294 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 23:58:41.246659 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260928 23:58:41.247927 5435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } }
I20260928 23:58:41.248237 5435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:41.248354 5435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ebf3a1bb7e45eb915e23cdd0ce80a4, State: Initialized, Role: FOLLOWER
I20260928 23:58:41.248862 5435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } }
I20260928 23:58:41.249035 5435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:41.249130 5435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:41.249233 5435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:41.250367 5435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } }
I20260928 23:58:41.250722 5435 leader_election.cc:302] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ebf3a1bb7e45eb915e23cdd0ce80a4; no voters:
I20260928 23:58:41.251065 5435 leader_election.cc:288] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:41.251260 5440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:41.251523 5440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [term 1 LEADER]: Becoming Leader. State: Replica: 24ebf3a1bb7e45eb915e23cdd0ce80a4, State: Running, Role: LEADER
I20260928 23:58:41.251933 5440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } }
I20260928 23:58:41.252440 5435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:41.253778 5442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ebf3a1bb7e45eb915e23cdd0ce80a4. Latest consensus state: current_term: 1 leader_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } } }
I20260928 23:58:41.253935 5442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:41.254366 5441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ebf3a1bb7e45eb915e23cdd0ce80a4" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 41077 } } }
I20260928 23:58:41.254506 5441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4 [sys.catalog]: This master's current role is: LEADER
W20260928 23:58:41.257516 5455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:58:41.257664 5455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:58:41.257767 5451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:41.258838 5451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:41.264286 5451 catalog_manager.cc:1347] Generated new cluster ID: 6f542939156c491b912f5092b7c5d3c3
I20260928 23:58:41.264375 5451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:41.280653 5451 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:41.281731 5451 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:41.288789 5451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: Generated new TSK 0
I20260928 23:58:41.289781 5451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:58:41.335587 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:41.470144 5461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:41.470441 5461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:41.470486 5461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:41.474872 5461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:41.474959 5461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:41.475023 5461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:41.480767 5461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:41.481992 5461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:41.483114 5461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:41.493690 5469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.493690 5466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.493739 5467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.494017 5461 server_base.cc:1034] running on GCE node
I20260928 23:58:41.494621 5461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:41.495383 5461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:41.496585 5461 hybrid_clock.cc:648] HybridClock initialized: now 1790639921496543 us; error 56 us; skew 500 ppm
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:41.499760 5461 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:58:41.500815 5461 webserver.cc:460] Webserver started at http://127.20.88.1:42885/ using document root <none> and password file <none>
I20260928 23:58:41.501394 5461 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:41.501461 5461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:41.501642 5461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:41.503993 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "3b69a66f1764400096b60b652af1eaf7"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "aa1bffa375867fde36a9d9d04bd5ce4d"
server_key_iv: "ffb91152d34296af07cf21c6be5dc8cc"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.504670 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "3b69a66f1764400096b60b652af1eaf7"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "aa1bffa375867fde36a9d9d04bd5ce4d"
server_key_iv: "ffb91152d34296af07cf21c6be5dc8cc"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.508848 5461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:41.511538 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.513088 5461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:41.513209 5461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "3b69a66f1764400096b60b652af1eaf7"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "aa1bffa375867fde36a9d9d04bd5ce4d"
server_key_iv: "ffb91152d34296af07cf21c6be5dc8cc"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.513381 5461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:41.526404 5461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:41.530309 5461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:41.530485 5461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:41.531072 5461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:41.531914 5461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:41.531963 5461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.532004 5461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:41.532027 5461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.541232 5461 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39819
I20260928 23:58:41.541297 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39819 every 8 connection(s)
I20260928 23:58:41.542220 5461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:41.544189 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5461
I20260928 23:58:41.544350 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:41.544656 20832 external_mini_cluster.cc:1428] Setting key 8031d5895fac55f41c83f3fa61ffe467
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:58:41.557566 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:41.544200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:39461 (local address 127.20.88.62:41077)
0928 23:58:41.544397 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:41.544402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:41.545801 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 23:58:41.546798 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:41.546804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:41.546870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:41.546973 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:41.548264 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.548841 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.550052 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.550315 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.550438 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:41.553246 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:41.553249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:41.553251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:41.553280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:41.554951 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.555605 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.555608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.555610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.555700 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.556139 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.556142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.556143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.556424 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:41.556536 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:41.557250 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 23:58:41.557393 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 23:58:41.558550 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:41.544692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41077 (local address 127.20.88.1:39461)
0928 23:58:41.545646 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:41.545704 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:41.546588 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:41.547144 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:41.547151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:41.547527 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:41.548070 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:41.548087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.548981 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.548985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:41.549909 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:41.549919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.550492 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.550496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:41.550678 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:41.551333 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:41.551351 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:41.553047 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:41.555124 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:41.555131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:41.555153 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:41.555492 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:41.555794 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:41.555797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:41.555799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:41.555963 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:41.556557 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:41.556563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:41.556969 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 23:58:41.557195 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 23:58:41.557566 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":568,"threads_started":1}
I20260928 23:58:41.559918 5590 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41077
I20260928 23:58:41.560204 5590 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:41.560866 5590 heartbeater.cc:511] Master 127.20.88.62:41077 requested a full tablet report, sending...
I20260928 23:58:41.562683 5399 ts_manager.cc:194] Registered new tserver with Master: 3b69a66f1764400096b60b652af1eaf7 (127.20.88.1:39819)
I20260928 23:58:41.563953 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:39461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:58:41.576871 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:41.695235 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:41.695552 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:41.695597 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:41.699853 5595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:41.699940 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:41.700044 5595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:41.705749 5595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:41.706918 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:41.708033 5595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:41.718186 5600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.718210 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.718204 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.718436 5595 server_base.cc:1034] running on GCE node
I20260928 23:58:41.718894 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:41.719514 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:41.720675 5595 hybrid_clock.cc:648] HybridClock initialized: now 1790639921720656 us; error 33 us; skew 500 ppm
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:41.723557 5595 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:58:41.724511 5595 webserver.cc:460] Webserver started at http://127.20.88.2:46353/ using document root <none> and password file <none>
I20260928 23:58:41.725073 5595 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:41.725134 5595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:41.725301 5595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:41.727368 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "4ae124a9ac364a0f93705172c0a44250"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "7482dccc8d10a63296230b6dde50c081"
server_key_iv: "2f5e703b5f9b2891ffcc108382fb19e1"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.727902 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "4ae124a9ac364a0f93705172c0a44250"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "7482dccc8d10a63296230b6dde50c081"
server_key_iv: "2f5e703b5f9b2891ffcc108382fb19e1"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.732085 5595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 23:58:41.734884 5610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.736493 5595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:41.736649 5595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "4ae124a9ac364a0f93705172c0a44250"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "7482dccc8d10a63296230b6dde50c081"
server_key_iv: "2f5e703b5f9b2891ffcc108382fb19e1"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.736790 5595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:41.747736 5595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:41.751569 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:41.751808 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:41.752589 5595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:41.753641 5595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:41.753698 5595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.753744 5595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:41.753788 5595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.763335 5595 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:43643
I20260928 23:58:41.763355 5723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:43643 every 8 connection(s)
I20260928 23:58:41.764430 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:41.774313 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5595
I20260928 23:58:41.774466 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:41.774755 20832 external_mini_cluster.cc:1428] Setting key 5ea8f6e6a73a8c18bc092147f47aeaab
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:41.780668 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:41.766396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:46975 (local address 127.20.88.62:41077)
0928 23:58:41.766540 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:41.766544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:41.767441 (+ 897us) server_negotiation.cc:407] Connection header received
0928 23:58:41.768898 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:41.768902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:41.768975 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:41.769065 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:41.770312 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.770881 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.772215 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.772502 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.772628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:41.775956 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:41.775960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:41.775963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:41.775992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:41.777915 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.778630 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.778638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.778641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.778760 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:41.779262 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:41.779265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:41.779266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:41.779524 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:41.779618 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:41.780409 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 23:58:41.780538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:58:41.781994 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:41.766814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41077 (local address 127.20.88.2:46975)
0928 23:58:41.767285 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:41.767331 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:41.768674 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:41.769240 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:41.769248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:41.769625 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:41.770156 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:41.770171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.771027 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.771030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:41.772071 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:41.772083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:41.772682 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:41.772686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:41.772917 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:41.773588 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:41.773606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:41.775769 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:41.778060 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:41.778067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:41.778085 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:41.778392 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:41.778914 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:41.778919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:41.778922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:41.779154 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:41.779640 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:41.779649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:41.780048 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 23:58:41.780325 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 23:58:41.780747 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260928 23:58:41.783411 5724 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41077
I20260928 23:58:41.783696 5724 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:41.784279 5724 heartbeater.cc:511] Master 127.20.88.62:41077 requested a full tablet report, sending...
I20260928 23:58:41.785472 5399 ts_manager.cc:194] Registered new tserver with Master: 4ae124a9ac364a0f93705172c0a44250 (127.20.88.2:43643)
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:58:41.786384 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:46975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:58:41.808468 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:41.925196 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:41.925498 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:41.925544 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:41.929852 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:41.929939 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:41.930002 5729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:41.935711 5729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:41077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:41.936949 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:41.938005 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:41.948411 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:41.948464 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.948653 5729 server_base.cc:1034] running on GCE node
W20260928 23:58:41.948427 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:41.949218 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:41.949961 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:41.951155 5729 hybrid_clock.cc:648] HybridClock initialized: now 1790639921951137 us; error 35 us; skew 500 ppm
Sep 28 23:58:41 dist-test-slave-wvpw krb5kdc[5354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:41.954383 5729 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:58:41.955438 5729 webserver.cc:460] Webserver started at http://127.20.88.3:33681/ using document root <none> and password file <none>
I20260928 23:58:41.956066 5729 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:41.956135 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:41.956400 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:41.958590 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "00a4ec09831e4d8383f6c43b30bd2f52"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "e4bddb79f7a68a926e8c6440a983fda9"
server_key_iv: "f8bc9eeb0681b794f788e6a7cfc262d4"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.959223 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "00a4ec09831e4d8383f6c43b30bd2f52"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "e4bddb79f7a68a926e8c6440a983fda9"
server_key_iv: "f8bc9eeb0681b794f788e6a7cfc262d4"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.963399 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:58:41.966144 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.967550 5729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:41.967708 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "00a4ec09831e4d8383f6c43b30bd2f52"
format_stamp: "Formatted at 2026-09-28 23:58:41 on dist-test-slave-wvpw"
server_key: "e4bddb79f7a68a926e8c6440a983fda9"
server_key_iv: "f8bc9eeb0681b794f788e6a7cfc262d4"
server_key_version: "encryptionkey@0"
I20260928 23:58:41.967841 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:41.988613 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:41.992136 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:41.992373 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:41.993002 5729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:41.993870 5729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:41.994026 5729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:41.994146 5729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:41.994196 5729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.003364 5729 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:36013
I20260928 23:58:42.003382 5857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:36013 every 8 connection(s)
I20260928 23:58:42.004379 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:42.005628 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5729
I20260928 23:58:42.005784 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:42.006098 20832 external_mini_cluster.cc:1428] Setting key ce97f153dd8ca0b844a64e6a83a9d783
Sep 28 23:58:42 dist-test-slave-wvpw krb5kdc[5354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639921, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:42.020037 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.006463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:41265 (local address 127.20.88.62:41077)
0928 23:58:42.006656 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.006664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.007669 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 23:58:42.009360 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.009363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.009427 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.009509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:42.010822 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.011414 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.012702 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.012955 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.013086 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.015828 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:42.015831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:42.015833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:42.015874 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:42.017636 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:42.018159 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:42.018162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:42.018163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:42.018266 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:42.018637 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:42.018640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:42.018642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:42.018889 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:42.018999 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.019699 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.019851 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:58:42.021004 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.006922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:41077 (local address 127.20.88.3:41265)
0928 23:58:42.007519 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.007584 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.009102 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.009727 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.009739 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.010151 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:42.010671 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.010686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.011555 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.011559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.012569 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.012580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.013134 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.013139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.013331 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.014022 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:42.014041 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:42.015657 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:42.017742 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:42.017746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:42.017760 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:42.018068 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:42.018365 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:42.018367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:42.018368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:42.018532 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:42.019016 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:42.019024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:42.019407 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.019642 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.020005 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":130,"threads_started":1}
I20260928 23:58:42.022279 5858 heartbeater.cc:347] Connected to a master server at 127.20.88.62:41077
I20260928 23:58:42.022568 5858 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:42.023098 5858 heartbeater.cc:511] Master 127.20.88.62:41077 requested a full tablet report, sending...
I20260928 23:58:42.024158 5399 ts_manager.cc:194] Registered new tserver with Master: 00a4ec09831e4d8383f6c43b30bd2f52 (127.20.88.3:36013)
I20260928 23:58:42.024798 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:41265
I20260928 23:58:42.032164 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:42.041739 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.034249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.20.88.62:41077)
0928 23:58:42.034378 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.034382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.034551 (+ 169us) server_negotiation.cc:407] Connection header received
0928 23:58:42.034658 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.034660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.034702 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.034773 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:42.035494 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.036014 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.037011 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.037239 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.037333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.038697 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:42.038700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:42.038702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:42.038729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:42.040066 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:42.040605 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:42.040608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:42.040609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:42.040718 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:42.040964 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:42.040967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:42.040968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:42.041201 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:42.041332 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.041486 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.041588 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:58:42.045020 5399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:42.048105 5399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:42.063812 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.057662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:43643 (local address 127.0.0.1:37262)
0928 23:58:42.058830 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.058843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.058933 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.059532 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.059535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.059566 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:42.059767 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.059774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061464 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.061466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.062391 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.062400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.063332 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.063335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.063428 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.063461 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.063566 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.063665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"mutex_wait_us":84,"spinlock_wait_cycles":4096,"thread_start_us":121,"threads_started":1}
I20260928 23:58:42.063889 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.058216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.20.88.2:43643)
0928 23:58:42.058675 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.058689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.059026 (+ 337us) server_negotiation.cc:407] Connection header received
0928 23:58:42.059102 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.059106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.059247 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.059364 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:42.060325 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061307 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.062521 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.063171 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.063284 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.063447 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.063614 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.063719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":110,"threads_started":1}
I20260928 23:58:42.064996 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.057889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39819 (local address 127.0.0.1:49288)
0928 23:58:42.058833 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.058843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.058954 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.059533 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.059535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.059564 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:42.059767 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.059774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061201 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.061204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.062914 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.062925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.064571 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.064575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.064666 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.064689 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.064761 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.064849 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":286,"spinlock_wait_cycles":5248,"thread_start_us":103,"threads_started":1}
I20260928 23:58:42.065492 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.058310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.20.88.1:39819)
0928 23:58:42.058711 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.058723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.059100 (+ 377us) server_negotiation.cc:407] Connection header received
0928 23:58:42.059170 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.059174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.059354 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.059499 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:42.059888 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061082 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.063048 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.063840 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.063986 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.064130 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.065246 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.065355 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":109,"threads_started":1}
I20260928 23:58:42.066018 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.057423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:36013 (local address 127.0.0.1:58354)
0928 23:58:42.058256 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.058311 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.058446 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.059355 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.059361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.059384 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:42.059562 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.059570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061182 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.061187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.062914 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.062925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.065564 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.065569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.065671 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.065695 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.065771 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.065846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":170,"thread_start_us":86,"threads_started":1}
I20260928 23:58:42.066084 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.057579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.20.88.3:36013)
0928 23:58:42.058804 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.058815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.058839 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:58:42.058925 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.058931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.059189 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.059353 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:42.059674 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.061005 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.064216 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.065021 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.065304 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.065475 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.065820 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.065925 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":193,"threads_started":1}
I20260928 23:58:42.067195 5658 tablet_service.cc:1467] Processing CreateTablet for tablet e54962c2df3d4a8697ab277a3aa25508 (DEFAULT_TABLE table=test-table [id=5f1ff28b3be8422ba391e60f6c046827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:42.067852 5524 tablet_service.cc:1467] Processing CreateTablet for tablet e54962c2df3d4a8697ab277a3aa25508 (DEFAULT_TABLE table=test-table [id=5f1ff28b3be8422ba391e60f6c046827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:42.068198 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54962c2df3d4a8697ab277a3aa25508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:42.068738 5524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54962c2df3d4a8697ab277a3aa25508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:42.074855 5875 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Bootstrap starting.
I20260928 23:58:42.074608 5792 tablet_service.cc:1467] Processing CreateTablet for tablet e54962c2df3d4a8697ab277a3aa25508 (DEFAULT_TABLE table=test-table [id=5f1ff28b3be8422ba391e60f6c046827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:42.075817 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54962c2df3d4a8697ab277a3aa25508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:42.076254 5876 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: Bootstrap starting.
I20260928 23:58:42.077250 5875 tablet_bootstrap.cc:654] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:42.078333 5875 log.cc:824] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:42.078383 5876 tablet_bootstrap.cc:654] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:42.079231 5876 log.cc:824] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:42.080538 5875 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: No bootstrap required, opened a new log
I20260928 23:58:42.080747 5875 ts_tablet_manager.cc:1397] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:58:42.081506 5876 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: No bootstrap required, opened a new log
I20260928 23:58:42.081699 5876 ts_tablet_manager.cc:1397] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:42.083634 5879 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Bootstrap starting.
I20260928 23:58:42.084055 5875 raft_consensus.cc:348] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.084408 5875 raft_consensus.cc:374] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:42.084477 5875 raft_consensus.cc:729] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae124a9ac364a0f93705172c0a44250, State: Initialized, Role: FOLLOWER
I20260928 23:58:42.085086 5875 consensus_queue.cc:260] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.086150 5875 ts_tablet_manager.cc:1428] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:58:42.086519 5724 heartbeater.cc:503] Master 127.20.88.62:41077 was elected leader, sending a full tablet report...
I20260928 23:58:42.086546 5879 tablet_bootstrap.cc:654] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:42.086445 5876 raft_consensus.cc:348] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.086741 5876 raft_consensus.cc:374] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:42.086805 5876 raft_consensus.cc:729] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b69a66f1764400096b60b652af1eaf7, State: Initialized, Role: FOLLOWER
I20260928 23:58:42.087249 5876 consensus_queue.cc:260] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.087874 5590 heartbeater.cc:503] Master 127.20.88.62:41077 was elected leader, sending a full tablet report...
I20260928 23:58:42.088282 5876 ts_tablet_manager.cc:1428] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:58:42.087889 5879 log.cc:824] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:42.089236 5880 raft_consensus.cc:482] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:42.089413 5880 raft_consensus.cc:504] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.090675 5880 leader_election.cc:288] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a4ec09831e4d8383f6c43b30bd2f52 (127.20.88.3:36013), 3b69a66f1764400096b60b652af1eaf7 (127.20.88.1:39819)
I20260928 23:58:42.092005 5879 tablet_bootstrap.cc:492] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: No bootstrap required, opened a new log
I20260928 23:58:42.092366 5879 ts_tablet_manager.cc:1397] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260928 23:58:42.095681 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.091236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:57583 (local address 127.20.88.1:39819)
0928 23:58:42.091354 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.091359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.091644 (+ 285us) server_negotiation.cc:407] Connection header received
0928 23:58:42.091833 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.091835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.091887 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.091934 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:42.092380 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.093073 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.094517 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.094957 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.095020 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.095071 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.095465 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.095546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:58:42.095681 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.091074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:36013 (local address 127.20.88.2:35463)
0928 23:58:42.091171 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.091186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.091268 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.091704 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.091708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.091728 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:42.091842 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.091849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.092939 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.092943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.094180 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.094193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.095170 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.095174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.095291 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.095359 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.095441 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.095546 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":47}
I20260928 23:58:42.095728 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.091262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:35463 (local address 127.20.88.3:36013)
0928 23:58:42.091367 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.091373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.091388 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:58:42.091449 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.091452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.091507 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.091585 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:42.091970 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.092807 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.094329 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.094938 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.095043 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.095106 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.095531 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.095609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:58:42.096146 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.091074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39819 (local address 127.20.88.2:57583)
0928 23:58:42.091565 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.091578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.091714 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.092076 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.092079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.092097 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:42.092224 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.092250 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.093211 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.093214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.094350 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.094375 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.095224 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.095227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.095306 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.095346 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.095448 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.096020 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":161,"thread_start_us":113,"threads_started":1}
I20260928 23:58:42.096369 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54962c2df3d4a8697ab277a3aa25508" candidate_uuid: "4ae124a9ac364a0f93705172c0a44250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" is_pre_election: true
I20260928 23:58:42.096801 5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54962c2df3d4a8697ab277a3aa25508" candidate_uuid: "4ae124a9ac364a0f93705172c0a44250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b69a66f1764400096b60b652af1eaf7" is_pre_election: true
I20260928 23:58:42.097115 5544 raft_consensus.cc:2454] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae124a9ac364a0f93705172c0a44250 in term 0.
W20260928 23:58:42.097282 5613 leader_election.cc:341] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00a4ec09831e4d8383f6c43b30bd2f52 (127.20.88.3:36013): Illegal state: must be running to vote when last-logged opid is not known
I20260928 23:58:42.097568 5614 leader_election.cc:302] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b69a66f1764400096b60b652af1eaf7, 4ae124a9ac364a0f93705172c0a44250; no voters: 00a4ec09831e4d8383f6c43b30bd2f52
I20260928 23:58:42.097621 5879 raft_consensus.cc:348] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.097846 5880 raft_consensus.cc:2781] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:42.097916 5879 raft_consensus.cc:374] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:42.097939 5880 raft_consensus.cc:482] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:42.097972 5879 raft_consensus.cc:729] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a4ec09831e4d8383f6c43b30bd2f52, State: Initialized, Role: FOLLOWER
I20260928 23:58:42.097988 5880 raft_consensus.cc:3037] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:42.098457 5879 consensus_queue.cc:260] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.099270 5858 heartbeater.cc:503] Master 127.20.88.62:41077 was elected leader, sending a full tablet report...
I20260928 23:58:42.099309 5880 raft_consensus.cc:504] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.099571 5879 ts_tablet_manager.cc:1428] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260928 23:58:42.099983 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54962c2df3d4a8697ab277a3aa25508" candidate_uuid: "4ae124a9ac364a0f93705172c0a44250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a4ec09831e4d8383f6c43b30bd2f52"
I20260928 23:58:42.100116 5812 raft_consensus.cc:3037] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:42.100145 5880 leader_election.cc:288] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [CANDIDATE]: Term 1 election: Requested vote from peers 00a4ec09831e4d8383f6c43b30bd2f52 (127.20.88.3:36013), 3b69a66f1764400096b60b652af1eaf7 (127.20.88.1:39819)
I20260928 23:58:42.100497 5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e54962c2df3d4a8697ab277a3aa25508" candidate_uuid: "4ae124a9ac364a0f93705172c0a44250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b69a66f1764400096b60b652af1eaf7"
I20260928 23:58:42.100621 5544 raft_consensus.cc:3037] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:42.101567 5812 raft_consensus.cc:2454] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae124a9ac364a0f93705172c0a44250 in term 1.
I20260928 23:58:42.101917 5613 leader_election.cc:302] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a4ec09831e4d8383f6c43b30bd2f52, 4ae124a9ac364a0f93705172c0a44250; no voters:
I20260928 23:58:42.101994 5544 raft_consensus.cc:2454] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae124a9ac364a0f93705172c0a44250 in term 1.
I20260928 23:58:42.102110 5880 raft_consensus.cc:2781] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:42.102339 5880 raft_consensus.cc:686] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae124a9ac364a0f93705172c0a44250, State: Running, Role: LEADER
I20260928 23:58:42.102720 5880 consensus_queue.cc:237] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } }
I20260928 23:58:42.105832 5399 catalog_manager.cc:5495] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae124a9ac364a0f93705172c0a44250 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "4ae124a9ac364a0f93705172c0a44250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae124a9ac364a0f93705172c0a44250" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 } health_report { overall_health: UNKNOWN } } }
I20260928 23:58:42.129068 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.123948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.20.88.2:43643)
0928 23:58:42.124125 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.124130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.124147 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:58:42.124381 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.124386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.124463 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.124584 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:42.124922 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.125766 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.127118 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.127492 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.127614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.127988 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:42.128674 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:42.128806 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.128875 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.128929 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:58:42.135222 5812 raft_consensus.cc:1261] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 1 FOLLOWER]: Refusing update from remote peer 4ae124a9ac364a0f93705172c0a44250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:42.136070 5544 raft_consensus.cc:1261] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 1 FOLLOWER]: Refusing update from remote peer 4ae124a9ac364a0f93705172c0a44250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:42.136024 5885 consensus_queue.cc:1035] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a4ec09831e4d8383f6c43b30bd2f52" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:42.136708 5885 consensus_queue.cc:1035] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b69a66f1764400096b60b652af1eaf7" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:42.145294 5888 mvcc.cc:205] Tried to move back new op lower bound from 7334461121058414592 to 7334461120935493632. Current Snapshot: MvccSnapshot[applied={T|T < 7334461121058414592}]
I20260928 23:58:42.147331 5889 mvcc.cc:205] Tried to move back new op lower bound from 7334461121058414592 to 7334461120935493632. Current Snapshot: MvccSnapshot[applied={T|T < 7334461121058414592}]
I20260928 23:58:42.155614 5399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:42.155823 5399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:42.158831 5399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ebf3a1bb7e45eb915e23cdd0ce80a4: Sending DeleteTablet for 3 replicas of tablet e54962c2df3d4a8697ab277a3aa25508
I20260928 23:58:42.159852 5658 tablet_service.cc:1514] Processing DeleteTablet for tablet e54962c2df3d4a8697ab277a3aa25508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:37262
I20260928 23:58:42.159991 5792 tablet_service.cc:1514] Processing DeleteTablet for tablet e54962c2df3d4a8697ab277a3aa25508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:58354
I20260928 23:58:42.159981 5524 tablet_service.cc:1514] Processing DeleteTablet for tablet e54962c2df3d4a8697ab277a3aa25508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:42 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:49288
I20260928 23:58:42.160735 5899 tablet_replica.cc:330] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7: stopping tablet replica
I20260928 23:58:42.160735 5898 tablet_replica.cc:330] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: stopping tablet replica
I20260928 23:58:42.161139 5898 raft_consensus.cc:2229] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:42.161139 5899 raft_consensus.cc:2229] T e54962c2df3d4a8697ab277a3aa25508 P 3b69a66f1764400096b60b652af1eaf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:42.161360 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5461
I20260928 23:58:42.161566 5898 raft_consensus.cc:2258] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:42.162047 5897 tablet_replica.cc:330] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: stopping tablet replica
I20260928 23:58:42.162369 5897 raft_consensus.cc:2229] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:42.162709 5897 raft_consensus.cc:2258] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:42.164736 5897 ts_tablet_manager.cc:1905] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:42.165506 5898 ts_tablet_manager.cc:1905] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:42.168000 5897 ts_tablet_manager.cc:1918] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:42.168126 5897 log.cc:1196] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/e54962c2df3d4a8697ab277a3aa25508
I20260928 23:58:42.168618 5897 ts_tablet_manager.cc:1939] T e54962c2df3d4a8697ab277a3aa25508 P 00a4ec09831e4d8383f6c43b30bd2f52: Deleting consensus metadata
I20260928 23:58:42.168673 5898 ts_tablet_manager.cc:1918] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:42.168795 5898 log.cc:1196] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/e54962c2df3d4a8697ab277a3aa25508
I20260928 23:58:42.169258 5898 ts_tablet_manager.cc:1939] T e54962c2df3d4a8697ab277a3aa25508 P 4ae124a9ac364a0f93705172c0a44250: Deleting consensus metadata
I20260928 23:58:42.170460 5385 catalog_manager.cc:4841] TS 00a4ec09831e4d8383f6c43b30bd2f52 (127.20.88.3:36013): tablet e54962c2df3d4a8697ab277a3aa25508 (table test-table [id=5f1ff28b3be8422ba391e60f6c046827]) successfully deleted
I20260928 23:58:42.170519 5383 catalog_manager.cc:4841] TS 4ae124a9ac364a0f93705172c0a44250 (127.20.88.2:43643): tablet e54962c2df3d4a8697ab277a3aa25508 (table test-table [id=5f1ff28b3be8422ba391e60f6c046827]) successfully deleted
W20260928 23:58:42.173897 5386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:39819 (error 108)
I20260928 23:58:42.174870 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5595
I20260928 23:58:42.175060 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.174736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39819 (local address 127.0.0.1:49298)
0928 23:58:42.174884 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.174958 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:58:42.176388 5386 catalog_manager.cc:4584] TS 3b69a66f1764400096b60b652af1eaf7 (127.20.88.1:39819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e54962c2df3d4a8697ab277a3aa25508: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39819: connect: Connection refused (error 111)
I20260928 23:58:42.181020 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5729
I20260928 23:58:42.187470 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5367
2026-09-28T23:58:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2338 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T23:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:42Z Disabled control of system clock
I20260928 23:58:42.220846 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43903
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:43903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:42.339022 5904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:42.339355 5904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:42.339395 5904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:42.343719 5904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:42.343808 5904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:42.343824 5904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:42.343842 5904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:42.343858 5904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:42.349093 5904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:43903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:42.350315 5904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:42.351775 5904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:42.361357 5909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:42.361370 5904 server_base.cc:1034] running on GCE node
W20260928 23:58:42.361370 5910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:42.361383 5912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:42.362200 5904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:42.363511 5904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:42.364711 5904 hybrid_clock.cc:648] HybridClock initialized: now 1790639922364658 us; error 69 us; skew 500 ppm
I20260928 23:58:42.366846 5904 webserver.cc:460] Webserver started at http://127.20.88.62:33273/ using document root <none> and password file <none>
I20260928 23:58:42.367511 5904 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:42.367611 5904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:42.367869 5904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:42.370138 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "29dbaa07743b423f922ca93a14080089"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "43f9669d2ee25d20084fbb59fc861515"
server_key_iv: "8030c33a2068e079e31224157d423ae5"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.370752 5904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "29dbaa07743b423f922ca93a14080089"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "43f9669d2ee25d20084fbb59fc861515"
server_key_iv: "8030c33a2068e079e31224157d423ae5"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.374951 5904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:42.377852 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.379298 5904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:42.379487 5904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "29dbaa07743b423f922ca93a14080089"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "43f9669d2ee25d20084fbb59fc861515"
server_key_iv: "8030c33a2068e079e31224157d423ae5"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.379637 5904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:42.395787 5904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:42.396684 5904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:42.396916 5904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:42.404341 5904 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:43903
I20260928 23:58:42.404366 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:43903 every 8 connection(s)
I20260928 23:58:42.405340 5904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:42.407858 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5904
I20260928 23:58:42.408007 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:42.408350 20832 external_mini_cluster.cc:1428] Setting key 69d34cb704c8770a22659173d6ac3f3f
I20260928 23:58:42.408811 5971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:42.416843 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: Bootstrap starting.
I20260928 23:58:42.418792 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.409875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.20.88.62:43903)
0928 23:58:42.410524 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.410541 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.410576 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:58:42.411577 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.411598 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.412035 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.412413 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:42.412769 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.413747 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.415126 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.415632 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.415754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.416820 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:42.416826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:42.416839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:42.417320 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:42.417381 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:42.417402 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:42.417484 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.417740 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.418053 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":147,"threads_started":1}
I20260928 23:58:42.419397 5971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:42.420302 5971 log.cc:824] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:42.422456 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: No bootstrap required, opened a new log
I20260928 23:58:42.426508 5971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } }
I20260928 23:58:42.426756 5971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:42.426791 5971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29dbaa07743b423f922ca93a14080089, State: Initialized, Role: FOLLOWER
I20260928 23:58:42.427210 5971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } }
I20260928 23:58:42.427337 5971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:42.427385 5971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:42.427453 5971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:42.428591 5971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } }
I20260928 23:58:42.428951 5971 leader_election.cc:302] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29dbaa07743b423f922ca93a14080089; no voters:
I20260928 23:58:42.429368 5971 leader_election.cc:288] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:42.429517 5976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:42.429765 5976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [term 1 LEADER]: Becoming Leader. State: Replica: 29dbaa07743b423f922ca93a14080089, State: Running, Role: LEADER
I20260928 23:58:42.430073 5976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } }
I20260928 23:58:42.430645 5971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:42.431699 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29dbaa07743b423f922ca93a14080089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } } }
I20260928 23:58:42.431847 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:42.432119 5978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29dbaa07743b423f922ca93a14080089. Latest consensus state: current_term: 1 leader_uuid: "29dbaa07743b423f922ca93a14080089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29dbaa07743b423f922ca93a14080089" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43903 } } }
I20260928 23:58:42.432217 5978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:42.432322 5985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:42.434974 5985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:42.441783 5985 catalog_manager.cc:1347] Generated new cluster ID: 1834814aaa974dd594a636d73d05b0e4
I20260928 23:58:42.441884 5985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:42.471278 5985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:42.472415 5985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:42.480372 5985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: Generated new TSK 0
I20260928 23:58:42.481395 5985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:42.484453 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--builtin_ntp_servers=127.20.88.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:42.606278 5995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:42.606601 5995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:42.606644 5995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:42.610883 5995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:42.610955 5995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:42.611016 5995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:42.616137 5995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:42.617434 5995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:42.619012 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:42.628965 6001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:42.628955 6003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:42.628976 6000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:42.629631 5995 server_base.cc:1034] running on GCE node
I20260928 23:58:42.630110 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:42.630812 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:42.631997 5995 hybrid_clock.cc:648] HybridClock initialized: now 1790639922631975 us; error 40 us; skew 500 ppm
I20260928 23:58:42.634115 5995 webserver.cc:460] Webserver started at http://127.20.88.1:44909/ using document root <none> and password file <none>
I20260928 23:58:42.634761 5995 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:42.634855 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:42.635093 5995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:42.637373 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "a6f69b22aaca4dfaba67ddce3761a278"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "6d974bdc3ba630e15ecc8738cc01ad1c"
server_key_iv: "381d21306aa6d404ac299f73148e624a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.638005 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "a6f69b22aaca4dfaba67ddce3761a278"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "6d974bdc3ba630e15ecc8738cc01ad1c"
server_key_iv: "381d21306aa6d404ac299f73148e624a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.642117 5995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:42.644850 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.646148 5995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:42.646297 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "a6f69b22aaca4dfaba67ddce3761a278"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "6d974bdc3ba630e15ecc8738cc01ad1c"
server_key_iv: "381d21306aa6d404ac299f73148e624a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.646435 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:42.661612 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:42.662369 5995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:42.662583 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:42.663192 5995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:42.664165 5995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:42.664234 5995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.664323 5995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:42.664366 5995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.673519 5995 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:33181
I20260928 23:58:42.673622 6122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:33181 every 8 connection(s)
I20260928 23:58:42.674566 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:42.682003 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 5995
I20260928 23:58:42.682152 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:42.682446 20832 external_mini_cluster.cc:1428] Setting key 47bd61f6118c1acb74e6ad12e62b8736
I20260928 23:58:42.684684 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--builtin_ntp_servers=127.20.88.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:42.685139 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.676500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:51353 (local address 127.20.88.62:43903)
0928 23:58:42.676650 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.676655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.678097 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 23:58:42.679534 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.679544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.679602 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.679716 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:42.680762 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.681298 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.682676 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.682967 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.683091 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.684467 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:42.684471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:42.684474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:42.684493 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:42.684513 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:42.684521 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:42.684599 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.684822 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.684932 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:58:42.686049 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.676979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43903 (local address 127.20.88.1:51353)
0928 23:58:42.677916 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.677966 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.679297 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.679929 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.679937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.680026 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:42.680624 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.680639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.681439 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.681442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.682540 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.682554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.683163 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.683168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.683382 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.684125 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:42.684127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:42.684172 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:42.684299 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:42.684601 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:42.684613 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:42.684623 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.684766 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.685106 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260928 23:58:42.687435 6123 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43903
I20260928 23:58:42.687700 6123 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:42.688330 6123 heartbeater.cc:511] Master 127.20.88.62:43903 requested a full tablet report, sending...
I20260928 23:58:42.690158 5935 ts_manager.cc:194] Registered new tserver with Master: a6f69b22aaca4dfaba67ddce3761a278 (127.20.88.1:33181)
I20260928 23:58:42.692032 5935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.1:51353
W20260928 23:58:42.802701 6126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:42.803027 6126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:42.803069 6126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:42.807271 6126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:42.807348 6126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:42.807411 6126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:42.812557 6126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:42.813704 6126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:42.815114 6126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:42.825764 6134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:42.825727 6131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:42.825834 6126 server_base.cc:1034] running on GCE node
W20260928 23:58:42.825752 6132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:42.826419 6126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:42.827126 6126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:42.828336 6126 hybrid_clock.cc:648] HybridClock initialized: now 1790639922828312 us; error 34 us; skew 500 ppm
I20260928 23:58:42.830356 6126 webserver.cc:460] Webserver started at http://127.20.88.2:42169/ using document root <none> and password file <none>
I20260928 23:58:42.831055 6126 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:42.831128 6126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:42.831416 6126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:42.833916 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "7792896fdf83476f859f0c04e61928bc"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "168b297eb4ca3b5d4af4fc290a11a481"
server_key_iv: "c2a17617542bb493f7a30caead3c6c7a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.834571 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "7792896fdf83476f859f0c04e61928bc"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "168b297eb4ca3b5d4af4fc290a11a481"
server_key_iv: "c2a17617542bb493f7a30caead3c6c7a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.838865 6126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:42.841979 6140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.843510 6126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:42.843678 6126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "7792896fdf83476f859f0c04e61928bc"
format_stamp: "Formatted at 2026-09-28 23:58:42 on dist-test-slave-wvpw"
server_key: "168b297eb4ca3b5d4af4fc290a11a481"
server_key_iv: "c2a17617542bb493f7a30caead3c6c7a"
server_key_version: "encryptionkey@0"
I20260928 23:58:42.843827 6126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:42.854980 6126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:42.855763 6126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:42.855978 6126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:42.856681 6126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:42.857707 6126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:42.857779 6126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.857854 6126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:42.857893 6126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:42.867098 6126 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:41145
I20260928 23:58:42.867123 6253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:41145 every 8 connection(s)
I20260928 23:58:42.868074 6126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:42.872046 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6126
I20260928 23:58:42.872195 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:42.872519 20832 external_mini_cluster.cc:1428] Setting key 3ca103549ee0117760ded603203b8eab
I20260928 23:58:42.874606 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--builtin_ntp_servers=127.20.88.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:42.877875 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.869967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:56965 (local address 127.20.88.62:43903)
0928 23:58:42.870137 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:42.870142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:42.871223 (+ 1081us) server_negotiation.cc:407] Connection header received
0928 23:58:42.872707 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:42.872711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:42.872776 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:42.872881 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:42.874000 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.874580 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.875770 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.876004 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.876116 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.877364 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:42.877368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:42.877370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:42.877382 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:42.877403 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:42.877413 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:42.877488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:42.877660 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:58:42.877728 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:58:42.878902 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:42.870386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43903 (local address 127.20.88.2:56965)
0928 23:58:42.871061 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:42.871105 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:42.872477 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:42.873079 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:42.873086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:42.873178 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:42.873854 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.873870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.874713 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.874716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.875637 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:42.875646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:42.876160 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:42.876164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:42.876402 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:42.877074 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:42.877076 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:42.877119 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:42.877192 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:42.877501 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:42.877514 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:42.877520 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:58:42.877657 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 23:58:42.877975 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260928 23:58:42.880098 6254 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43903
I20260928 23:58:42.880427 6254 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:42.881182 6254 heartbeater.cc:511] Master 127.20.88.62:43903 requested a full tablet report, sending...
I20260928 23:58:42.882496 5935 ts_manager.cc:194] Registered new tserver with Master: 7792896fdf83476f859f0c04e61928bc (127.20.88.2:41145)
I20260928 23:58:42.883133 5935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.2:56965
W20260928 23:58:42.991413 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:42.991698 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:42.991737 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:42.995887 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:42.995967 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:42.996024 6257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:43.001094 6257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:43903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:43.002245 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:43.003669 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:43.013917 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:43.013926 6265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:43.013942 6262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:43.014380 6257 server_base.cc:1034] running on GCE node
I20260928 23:58:43.014830 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:43.015563 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:43.016752 6257 hybrid_clock.cc:648] HybridClock initialized: now 1790639923016713 us; error 55 us; skew 500 ppm
I20260928 23:58:43.019047 6257 webserver.cc:460] Webserver started at http://127.20.88.3:34839/ using document root <none> and password file <none>
I20260928 23:58:43.019706 6257 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:43.019774 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:43.020002 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:43.022327 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "97a27de2d35f4894bd717c397d35fec5"
format_stamp: "Formatted at 2026-09-28 23:58:43 on dist-test-slave-wvpw"
server_key: "b349daaf51ce7e80e072893db9614327"
server_key_iv: "be24b7b181f0860668c608cfb466af9d"
server_key_version: "encryptionkey@0"
I20260928 23:58:43.022964 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "97a27de2d35f4894bd717c397d35fec5"
format_stamp: "Formatted at 2026-09-28 23:58:43 on dist-test-slave-wvpw"
server_key: "b349daaf51ce7e80e072893db9614327"
server_key_iv: "be24b7b181f0860668c608cfb466af9d"
server_key_version: "encryptionkey@0"
I20260928 23:58:43.027288 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:43.030231 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:43.031950 6257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:43.032096 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "97a27de2d35f4894bd717c397d35fec5"
format_stamp: "Formatted at 2026-09-28 23:58:43 on dist-test-slave-wvpw"
server_key: "b349daaf51ce7e80e072893db9614327"
server_key_iv: "be24b7b181f0860668c608cfb466af9d"
server_key_version: "encryptionkey@0"
I20260928 23:58:43.032243 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:43.046555 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:43.047426 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:43.047636 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:43.048336 6257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:43.049360 6257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:43.049429 6257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:43.049499 6257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:43.049551 6257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:43.058806 6257 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:38549
I20260928 23:58:43.058830 6384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:38549 every 8 connection(s)
I20260928 23:58:43.059816 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:43.061774 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6257
I20260928 23:58:43.061913 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:43.062220 20832 external_mini_cluster.cc:1428] Setting key 9963f0857be454aaca58a317934b690d
I20260928 23:58:43.069593 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.061762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:53481 (local address 127.20.88.62:43903)
0928 23:58:43.061943 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.061948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.063174 (+ 1226us) server_negotiation.cc:407] Connection header received
0928 23:58:43.064940 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.064943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.064991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.065057 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:43.065992 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.066519 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.067663 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.067879 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.067971 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.069143 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:43.069146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:43.069148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:43.069161 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:43.069182 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:43.069190 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:43.069250 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.069410 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.069469 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:58:43.070544 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.062201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:43903 (local address 127.20.88.3:53481)
0928 23:58:43.063013 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.063076 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.064742 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.065237 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.065245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.065336 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:43.065865 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.065878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.066626 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.066628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.067546 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.067556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.068024 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.068026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.068175 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.068867 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:43.068869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:43.068915 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:43.068991 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:43.069261 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:43.069268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:43.069272 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.069369 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.069636 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":125,"threads_started":1}
I20260928 23:58:43.071846 6385 heartbeater.cc:347] Connected to a master server at 127.20.88.62:43903
I20260928 23:58:43.072109 6385 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:43.072724 6385 heartbeater.cc:511] Master 127.20.88.62:43903 requested a full tablet report, sending...
I20260928 23:58:43.073833 5935 ts_manager.cc:194] Registered new tserver with Master: 97a27de2d35f4894bd717c397d35fec5 (127.20.88.3:38549)
I20260928 23:58:43.074404 5935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.3:53481
I20260928 23:58:43.076342 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:43.082113 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.078414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.20.88.62:43903)
0928 23:58:43.078586 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.078590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.078805 (+ 215us) server_negotiation.cc:407] Connection header received
0928 23:58:43.078855 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.078857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.078907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.078985 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:43.079293 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.079806 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.080715 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.080945 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.081031 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.081748 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:43.081752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:43.081754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:43.081767 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:43.081787 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:43.081796 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:43.081871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.081937 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.081990 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 23:58:43.085232 5935 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:43.088402 5935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:43.104812 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.098250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:38549 (local address 127.0.0.1:33918)
0928 23:58:43.099584 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.099597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.099716 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.100214 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.100217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.100254 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:43.100334 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.100339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.101581 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.101586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.103014 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.103024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.104184 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.104188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.104343 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.104391 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.104487 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.104620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":43,"thread_start_us":120,"threads_started":1}
I20260928 23:58:43.104897 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.099036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.20.88.3:38549)
0928 23:58:43.099371 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.099383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.099693 (+ 310us) server_negotiation.cc:407] Connection header received
0928 23:58:43.099828 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.099836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.100061 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.100483 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:43.100593 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.101466 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.103155 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.104041 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.104180 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.104453 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.104540 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.104660 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":151,"threads_started":1}
I20260928 23:58:43.105005 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.098058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:41145 (local address 127.0.0.1:38862)
0928 23:58:43.098575 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.098629 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.098767 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.100089 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.100096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.100130 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:43.100322 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.100329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.102047 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.102051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.103341 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.103354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.104624 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.104627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.104711 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.104731 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.104800 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.104878 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":62,"thread_start_us":96,"threads_started":1}
I20260928 23:58:43.105629 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.098290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.20.88.2:41145)
0928 23:58:43.099039 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.099056 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.099421 (+ 365us) server_negotiation.cc:407] Connection header received
0928 23:58:43.099522 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.099528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.099782 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.099981 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:43.100468 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.101874 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.103516 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.104213 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.104339 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.104563 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.105342 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.105469 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":153,"threads_started":1}
I20260928 23:58:43.106619 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.099111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:33181 (local address 127.0.0.1:46070)
0928 23:58:43.100065 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.100082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.100204 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.100761 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.100764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.100782 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:43.100884 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.100890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.102731 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.102734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.103717 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.103727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.105623 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.105629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.106218 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.106244 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.106355 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.106448 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":1,"thread_start_us":124,"threads_started":1}
I20260928 23:58:43.106729 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.099586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.20.88.1:33181)
0928 23:58:43.099985 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.099999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.100198 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:58:43.100358 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.100364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.100591 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.100773 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:43.100986 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.102606 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.103860 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.104793 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.105887 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.106107 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.106407 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.106524 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":176,"threads_started":1}
I20260928 23:58:43.108466 6188 tablet_service.cc:1467] Processing CreateTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 (DEFAULT_TABLE table=test-table [id=ddc27c234e76430384ed1547f87a20e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:43.108466 6319 tablet_service.cc:1467] Processing CreateTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 (DEFAULT_TABLE table=test-table [id=ddc27c234e76430384ed1547f87a20e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:43.109680 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240fe8ad84e34632b2f42b18fe0c9211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:43.109680 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240fe8ad84e34632b2f42b18fe0c9211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:43.109699 6057 tablet_service.cc:1467] Processing CreateTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 (DEFAULT_TABLE table=test-table [id=ddc27c234e76430384ed1547f87a20e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:43.110816 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240fe8ad84e34632b2f42b18fe0c9211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:43.118314 6404 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Bootstrap starting.
I20260928 23:58:43.118564 6403 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Bootstrap starting.
I20260928 23:58:43.120643 6402 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Bootstrap starting.
I20260928 23:58:43.120847 6403 tablet_bootstrap.cc:654] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:43.120885 6404 tablet_bootstrap.cc:654] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:43.121737 6403 log.cc:824] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:43.121676 6404 log.cc:824] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:43.122928 6402 tablet_bootstrap.cc:654] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:43.123749 6402 log.cc:824] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:43.124065 6403 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: No bootstrap required, opened a new log
I20260928 23:58:43.124370 6404 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: No bootstrap required, opened a new log
I20260928 23:58:43.124372 6403 ts_tablet_manager.cc:1397] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:43.124576 6404 ts_tablet_manager.cc:1397] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:58:43.126034 6402 tablet_bootstrap.cc:492] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: No bootstrap required, opened a new log
I20260928 23:58:43.126219 6402 ts_tablet_manager.cc:1397] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:43.128435 6403 raft_consensus.cc:348] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.128768 6403 raft_consensus.cc:374] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:43.128834 6403 raft_consensus.cc:729] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792896fdf83476f859f0c04e61928bc, State: Initialized, Role: FOLLOWER
I20260928 23:58:43.128959 6404 raft_consensus.cc:348] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.129212 6404 raft_consensus.cc:374] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:43.129266 6404 raft_consensus.cc:729] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f69b22aaca4dfaba67ddce3761a278, State: Initialized, Role: FOLLOWER
I20260928 23:58:43.129364 6403 consensus_queue.cc:260] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.129760 6404 consensus_queue.cc:260] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.130257 6254 heartbeater.cc:503] Master 127.20.88.62:43903 was elected leader, sending a full tablet report...
I20260928 23:58:43.130617 6123 heartbeater.cc:503] Master 127.20.88.62:43903 was elected leader, sending a full tablet report...
I20260928 23:58:43.130704 6403 ts_tablet_manager.cc:1428] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:43.130771 6404 ts_tablet_manager.cc:1428] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:43.131440 6402 raft_consensus.cc:348] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.131686 6402 raft_consensus.cc:374] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:43.131742 6402 raft_consensus.cc:729] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a27de2d35f4894bd717c397d35fec5, State: Initialized, Role: FOLLOWER
I20260928 23:58:43.132176 6402 consensus_queue.cc:260] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.132928 6385 heartbeater.cc:503] Master 127.20.88.62:43903 was elected leader, sending a full tablet report...
I20260928 23:58:43.133208 6402 ts_tablet_manager.cc:1428] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 23:58:43.175854 6124 tablet.cc:2369] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:43.307344 6408 raft_consensus.cc:482] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:43.307626 6408 raft_consensus.cc:504] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.308997 6408 leader_election.cc:288] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a27de2d35f4894bd717c397d35fec5 (127.20.88.3:38549), a6f69b22aaca4dfaba67ddce3761a278 (127.20.88.1:33181)
W20260928 23:58:43.310882 6386 tablet.cc:2369] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:43.312903 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.309204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:38549 (local address 127.20.88.2:45729)
0928 23:58:43.309332 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.309350 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.309476 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.309867 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.309870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.309888 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:43.310035 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.310041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.310788 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.310792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.311696 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.311705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.312505 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.312508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.312590 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.312612 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.312671 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.312729 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260928 23:58:43.312943 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.309443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:45729 (local address 127.20.88.3:38549)
0928 23:58:43.309657 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.309661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.309673 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:58:43.309725 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.309727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.309764 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.309855 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:43.310161 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.310676 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.311830 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.312380 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.312488 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.312551 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.312715 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.312804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 23:58:43.313258 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.309304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:33181 (local address 127.20.88.2:47679)
0928 23:58:43.309677 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.309688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:43.309813 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:43.310089 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:43.310092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:43.310108 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:43.310235 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.310240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.311093 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.311097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.311961 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:43.311970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.312832 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.312835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:43.312940 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.312968 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:58:43.313042 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:58:43.313115 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260928 23:58:43.313416 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.309430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:47679 (local address 127.20.88.1:33181)
0928 23:58:43.309561 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.309565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.309774 (+ 209us) server_negotiation.cc:407] Connection header received
0928 23:58:43.309928 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.309937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.309985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.310069 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:43.310356 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.310952 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.312077 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.312699 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.312802 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.312855 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.313206 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.313286 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:58:43.313663 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240fe8ad84e34632b2f42b18fe0c9211" candidate_uuid: "7792896fdf83476f859f0c04e61928bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a27de2d35f4894bd717c397d35fec5" is_pre_election: true
I20260928 23:58:43.314003 6077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240fe8ad84e34632b2f42b18fe0c9211" candidate_uuid: "7792896fdf83476f859f0c04e61928bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f69b22aaca4dfaba67ddce3761a278" is_pre_election: true
I20260928 23:58:43.314052 6339 raft_consensus.cc:2454] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792896fdf83476f859f0c04e61928bc in term 0.
I20260928 23:58:43.314283 6077 raft_consensus.cc:2454] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792896fdf83476f859f0c04e61928bc in term 0.
I20260928 23:58:43.314618 6142 leader_election.cc:302] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7792896fdf83476f859f0c04e61928bc, 97a27de2d35f4894bd717c397d35fec5; no voters:
I20260928 23:58:43.314932 6408 raft_consensus.cc:2781] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:43.315011 6408 raft_consensus.cc:482] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:43.315057 6408 raft_consensus.cc:3037] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:43.316088 6408 raft_consensus.cc:504] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.316521 6408 leader_election.cc:288] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [CANDIDATE]: Term 1 election: Requested vote from peers 97a27de2d35f4894bd717c397d35fec5 (127.20.88.3:38549), a6f69b22aaca4dfaba67ddce3761a278 (127.20.88.1:33181)
I20260928 23:58:43.316859 6077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240fe8ad84e34632b2f42b18fe0c9211" candidate_uuid: "7792896fdf83476f859f0c04e61928bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f69b22aaca4dfaba67ddce3761a278"
I20260928 23:58:43.316897 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240fe8ad84e34632b2f42b18fe0c9211" candidate_uuid: "7792896fdf83476f859f0c04e61928bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a27de2d35f4894bd717c397d35fec5"
I20260928 23:58:43.316977 6077 raft_consensus.cc:3037] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:43.317041 6339 raft_consensus.cc:3037] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:43.318080 6077 raft_consensus.cc:2454] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792896fdf83476f859f0c04e61928bc in term 1.
I20260928 23:58:43.318171 6339 raft_consensus.cc:2454] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792896fdf83476f859f0c04e61928bc in term 1.
I20260928 23:58:43.318456 6141 leader_election.cc:302] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7792896fdf83476f859f0c04e61928bc, a6f69b22aaca4dfaba67ddce3761a278; no voters:
I20260928 23:58:43.318661 6408 raft_consensus.cc:2781] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:43.318933 6408 raft_consensus.cc:686] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 1 LEADER]: Becoming Leader. State: Replica: 7792896fdf83476f859f0c04e61928bc, State: Running, Role: LEADER
I20260928 23:58:43.319300 6408 consensus_queue.cc:237] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } }
I20260928 23:58:43.322886 5934 catalog_manager.cc:5495] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7792896fdf83476f859f0c04e61928bc (127.20.88.2). New cstate: current_term: 1 leader_uuid: "7792896fdf83476f859f0c04e61928bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7792896fdf83476f859f0c04e61928bc" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 41145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 } health_report { overall_health: UNKNOWN } } }
I20260928 23:58:43.351130 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.347182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.20.88.2:41145)
0928 23:58:43.347308 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:43.347312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:43.347325 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:58:43.347464 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:43.347467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:43.347527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:43.347623 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:43.348012 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.348612 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.349526 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:43.349746 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:43.349862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:43.350236 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:43.350705 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:43.350843 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:43.350904 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:58:43.350967 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:58:43.357431 6077 raft_consensus.cc:1261] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 1 FOLLOWER]: Refusing update from remote peer 7792896fdf83476f859f0c04e61928bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:43.357432 6339 raft_consensus.cc:1261] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 1 FOLLOWER]: Refusing update from remote peer 7792896fdf83476f859f0c04e61928bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:43.358225 6412 consensus_queue.cc:1035] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f69b22aaca4dfaba67ddce3761a278" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:43.358445 6408 consensus_queue.cc:1035] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a27de2d35f4894bd717c397d35fec5" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:43.375799 5935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:43.376017 5935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:43.378774 5935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29dbaa07743b423f922ca93a14080089: Sending DeleteTablet for 3 replicas of tablet 240fe8ad84e34632b2f42b18fe0c9211
I20260928 23:58:43.379696 6319 tablet_service.cc:1514] Processing DeleteTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:43 UTC) from {username='slave'} at 127.0.0.1:33918
I20260928 23:58:43.379678 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:43 UTC) from {username='slave'} at 127.0.0.1:38862
I20260928 23:58:43.380313 6425 tablet_replica.cc:330] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: stopping tablet replica
I20260928 23:58:43.380277 6057 tablet_service.cc:1514] Processing DeleteTablet for tablet 240fe8ad84e34632b2f42b18fe0c9211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:43 UTC) from {username='slave'} at 127.0.0.1:46070
I20260928 23:58:43.380445 6426 tablet_replica.cc:330] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: stopping tablet replica
I20260928 23:58:43.380790 6425 raft_consensus.cc:2229] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:43.380792 6426 raft_consensus.cc:2229] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:43.381141 6426 raft_consensus.cc:2258] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:43.381402 6427 tablet_replica.cc:330] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278: stopping tablet replica
I20260928 23:58:43.381513 6425 raft_consensus.cc:2258] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:43.381759 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5995
I20260928 23:58:43.381685 6427 raft_consensus.cc:2229] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:43.382117 6427 raft_consensus.cc:2258] T 240fe8ad84e34632b2f42b18fe0c9211 P a6f69b22aaca4dfaba67ddce3761a278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:43.382839 6426 ts_tablet_manager.cc:1905] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:43.384092 6425 ts_tablet_manager.cc:1905] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:43.385721 6426 ts_tablet_manager.cc:1918] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:43.385838 6426 log.cc:1196] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/240fe8ad84e34632b2f42b18fe0c9211
I20260928 23:58:43.386176 6426 ts_tablet_manager.cc:1939] T 240fe8ad84e34632b2f42b18fe0c9211 P 97a27de2d35f4894bd717c397d35fec5: Deleting consensus metadata
I20260928 23:58:43.387603 5920 catalog_manager.cc:4841] TS 97a27de2d35f4894bd717c397d35fec5 (127.20.88.3:38549): tablet 240fe8ad84e34632b2f42b18fe0c9211 (table test-table [id=ddc27c234e76430384ed1547f87a20e3]) successfully deleted
I20260928 23:58:43.388005 6425 ts_tablet_manager.cc:1918] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:43.388118 6425 log.cc:1196] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/240fe8ad84e34632b2f42b18fe0c9211
I20260928 23:58:43.388464 6425 ts_tablet_manager.cc:1939] T 240fe8ad84e34632b2f42b18fe0c9211 P 7792896fdf83476f859f0c04e61928bc: Deleting consensus metadata
I20260928 23:58:43.389827 5921 catalog_manager.cc:4841] TS 7792896fdf83476f859f0c04e61928bc (127.20.88.2:41145): tablet 240fe8ad84e34632b2f42b18fe0c9211 (table test-table [id=ddc27c234e76430384ed1547f87a20e3]) successfully deleted
W20260928 23:58:43.390347 5919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:33181 (error 108)
I20260928 23:58:43.391182 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6126
I20260928 23:58:43.391533 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:43.391199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:33181 (local address 127.0.0.1:46082)
0928 23:58:43.391341 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:43.391428 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:58:43.392306 5919 catalog_manager.cc:4584] TS a6f69b22aaca4dfaba67ddce3761a278 (127.20.88.1:33181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 240fe8ad84e34632b2f42b18fe0c9211: Network error: Client connection negotiation failed: client connection to 127.20.88.1:33181: connect: Connection refused (error 111)
I20260928 23:58:43.397389 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6257
I20260928 23:58:43.402876 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 5904
2026-09-28T23:58:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:43 dist-test-slave-wvpw krb5kdc[6431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:43 dist-test-slave-wvpw krb5kdc[6431](info): set up 2 sockets
Sep 28 23:58:43 dist-test-slave-wvpw krb5kdc[6431](info): commencing operation
krb5kdc: starting...
W20260928 23:58:44.457152 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:44 dist-test-slave-wvpw krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:58:44.596748 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35859
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:44.714705 6444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:44.715075 6444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:44.715155 6444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:44.719525 6444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:44.719648 6444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:44.719686 6444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:44.719723 6444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:44.719758 6444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:44.725700 6444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:44.726931 6444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:44.728087 6444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:44.737325 6450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:44.737318 6452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:44.737318 6449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:44.737563 6444 server_base.cc:1034] running on GCE node
I20260928 23:58:44.738212 6444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:44.739506 6444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:44.740708 6444 hybrid_clock.cc:648] HybridClock initialized: now 1790639924740685 us; error 38 us; skew 500 ppm
Sep 28 23:58:44 dist-test-slave-wvpw krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639924, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:44.743651 6444 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:58:44.744634 6444 webserver.cc:460] Webserver started at http://127.20.88.62:45499/ using document root <none> and password file <none>
I20260928 23:58:44.745189 6444 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:44.745256 6444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:44.745431 6444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:44.747578 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140"
format_stamp: "Formatted at 2026-09-28 23:58:44 on dist-test-slave-wvpw"
server_key: "8947fea6429d216e114724ebe8de66cf"
server_key_iv: "6b55e50424221017eeac2caa35ae4090"
server_key_version: "encryptionkey@0"
I20260928 23:58:44.748136 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140"
format_stamp: "Formatted at 2026-09-28 23:58:44 on dist-test-slave-wvpw"
server_key: "8947fea6429d216e114724ebe8de66cf"
server_key_iv: "6b55e50424221017eeac2caa35ae4090"
server_key_version: "encryptionkey@0"
I20260928 23:58:44.752059 6444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:58:44.754879 6459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:44.756374 6444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:44.756492 6444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140"
format_stamp: "Formatted at 2026-09-28 23:58:44 on dist-test-slave-wvpw"
server_key: "8947fea6429d216e114724ebe8de66cf"
server_key_iv: "6b55e50424221017eeac2caa35ae4090"
server_key_version: "encryptionkey@0"
I20260928 23:58:44.756599 6444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:44.777626 6444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:44.781195 6444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:44.781371 6444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:44.788650 6444 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:35859
I20260928 23:58:44.788674 6511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:35859 every 8 connection(s)
I20260928 23:58:44.789685 6444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:44.792694 6512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:44.794273 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6444
I20260928 23:58:44.794572 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:44.794981 20832 external_mini_cluster.cc:1428] Setting key a36dd48c68b70b443b6d0ec1c2f44ce5
I20260928 23:58:44.800701 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: Bootstrap starting.
I20260928 23:58:44.803642 6512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:44.804523 6512 log.cc:824] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:58:44 dist-test-slave-wvpw krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:44.807037 6512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: No bootstrap required, opened a new log
I20260928 23:58:44.811487 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:44.796632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.20.88.62:35859)
0928 23:58:44.797115 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:44.797142 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:44.797188 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:58:44.798067 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:44.798094 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:44.798572 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:44.798994 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:44.799752 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:44.801305 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:44.802424 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:44.803123 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:44.803300 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:44.805974 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:44.805981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:44.805999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:44.806035 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:44.808045 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:44.808552 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:44.808559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:44.808566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:44.808715 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:44.808997 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:44.809000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:44.809002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:44.809634 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:44.809879 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:44.810102 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 23:58:44.810439 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":118,"threads_started":1}
I20260928 23:58:44.811970 6512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } }
I20260928 23:58:44.812227 6512 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:44.812309 6512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a5c3aa258d4e75baeb1a85b4e9c140, State: Initialized, Role: FOLLOWER
I20260928 23:58:44.812831 6512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } }
I20260928 23:58:44.812999 6512 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:44.813066 6512 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:44.813164 6512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:44.814456 6512 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } }
I20260928 23:58:44.814795 6512 leader_election.cc:302] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a5c3aa258d4e75baeb1a85b4e9c140; no voters:
I20260928 23:58:44.815065 6512 leader_election.cc:288] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:44.815354 6517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:44.815563 6517 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [term 1 LEADER]: Becoming Leader. State: Replica: a8a5c3aa258d4e75baeb1a85b4e9c140, State: Running, Role: LEADER
I20260928 23:58:44.815876 6517 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } }
I20260928 23:58:44.816414 6512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:44.817644 6519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a5c3aa258d4e75baeb1a85b4e9c140. Latest consensus state: current_term: 1 leader_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } } }
I20260928 23:58:44.817893 6519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:44.818151 6518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a5c3aa258d4e75baeb1a85b4e9c140" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 35859 } } }
I20260928 23:58:44.818274 6518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:44.818467 6528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:44.821538 6528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:44.827081 6528 catalog_manager.cc:1347] Generated new cluster ID: 1fe5a645ac5d4c2ca2f7327c63dfb643
I20260928 23:58:44.827167 6528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:44.837248 6528 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:44.838658 6528 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:44.847335 6528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: Generated new TSK 0
I20260928 23:58:44.848351 6528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:58:44.881883 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:45.010006 6538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:45.010303 6538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:45.010351 6538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:45.014591 6538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:45.014672 6538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:45.014765 6538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:45.020891 6538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:45.022084 6538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:45.023180 6538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:45.034145 6546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.034121 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.034356 6543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:45.034781 6538 server_base.cc:1034] running on GCE node
I20260928 23:58:45.035297 6538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:45.036011 6538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:45.037184 6538 hybrid_clock.cc:648] HybridClock initialized: now 1790639925037130 us; error 64 us; skew 500 ppm
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:45.045730 6538 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:58:45.046852 6538 webserver.cc:460] Webserver started at http://127.20.88.1:38863/ using document root <none> and password file <none>
I20260928 23:58:45.047464 6538 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:45.047636 6538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:45.047900 6538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:45.050444 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "0efb94e447c84ad2825121076c465e85"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "9fb345f94b269648cb79764bbd7e1376"
server_key_iv: "3f5fcd4e35aef9d68dee76215b2d6936"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.051314 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "0efb94e447c84ad2825121076c465e85"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "9fb345f94b269648cb79764bbd7e1376"
server_key_iv: "3f5fcd4e35aef9d68dee76215b2d6936"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.055603 6538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:45.058374 6553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.059931 6538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:58:45.060075 6538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "0efb94e447c84ad2825121076c465e85"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "9fb345f94b269648cb79764bbd7e1376"
server_key_iv: "3f5fcd4e35aef9d68dee76215b2d6936"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.060200 6538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:45.087400 6538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:45.091125 6538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:45.091336 6538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:45.092096 6538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:45.093005 6538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:45.093077 6538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.093144 6538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:45.093191 6538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.102634 6538 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:34583
I20260928 23:58:45.102679 6666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:34583 every 8 connection(s)
I20260928 23:58:45.103600 6538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:45.110145 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6538
I20260928 23:58:45.110303 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:45.110606 20832 external_mini_cluster.cc:1428] Setting key b5996fd3610cbc62e1535c619754395c
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:45.118955 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.105547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:60383 (local address 127.20.88.62:35859)
0928 23:58:45.105700 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.105704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.106547 (+ 843us) server_negotiation.cc:407] Connection header received
0928 23:58:45.107991 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.107994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.108055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.108138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:45.109444 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.110028 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.111479 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.111720 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.111818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.114597 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:45.114601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:45.114609 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:45.114638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:45.116281 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.116968 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.116971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.116973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.117065 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.117506 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.117509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.117511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.117777 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:45.117883 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.118550 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.118730 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:58:45.119793 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.105936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35859 (local address 127.20.88.1:60383)
0928 23:58:45.106399 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.106444 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.107724 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.108361 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.108368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.108750 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:45.109270 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.109285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.110170 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.110174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.111354 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.111365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.111899 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.111905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.112125 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.112855 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:45.112874 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:45.114376 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:45.116443 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.116450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.116471 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.116855 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.117202 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.117205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.117207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.117400 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.117884 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:45.117893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:45.118191 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.118453 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.118776 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:58:45.121160 6667 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35859
I20260928 23:58:45.121451 6667 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:45.122001 6667 heartbeater.cc:511] Master 127.20.88.62:35859 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:58:45.123838 6476 ts_manager.cc:194] Registered new tserver with Master: 0efb94e447c84ad2825121076c465e85 (127.20.88.1:34583)
I20260928 23:58:45.125451 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:60383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:58:45.145031 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:45.264539 6672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:45.264889 6672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:45.264962 6672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:45.269232 6672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:45.269305 6672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:45.269394 6672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:45.275408 6672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:45.276607 6672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:45.277666 6672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:45.288072 6677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.288069 6680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.288115 6678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:45.288555 6672 server_base.cc:1034] running on GCE node
I20260928 23:58:45.288985 6672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:45.289716 6672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:45.290879 6672 hybrid_clock.cc:648] HybridClock initialized: now 1790639925290862 us; error 36 us; skew 500 ppm
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:45.294023 6672 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:58:45.295020 6672 webserver.cc:460] Webserver started at http://127.20.88.2:46067/ using document root <none> and password file <none>
I20260928 23:58:45.295609 6672 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:45.295673 6672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:45.295910 6672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:45.298085 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "9814355b10014f0fbbab83abe4c92d64"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "d3c91f4d02636436f35ebba454e61f64"
server_key_iv: "e231f4d4432bf63c8e62210f77f532c1"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.298671 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "9814355b10014f0fbbab83abe4c92d64"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "d3c91f4d02636436f35ebba454e61f64"
server_key_iv: "e231f4d4432bf63c8e62210f77f532c1"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.302739 6672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:45.305524 6687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.306963 6672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:45.307147 6672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "9814355b10014f0fbbab83abe4c92d64"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "d3c91f4d02636436f35ebba454e61f64"
server_key_iv: "e231f4d4432bf63c8e62210f77f532c1"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.307286 6672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:45.321599 6672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:45.325421 6672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:45.325639 6672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:45.326308 6672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:45.327224 6672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:45.327297 6672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.327366 6672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:45.327414 6672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.337018 6672 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:39567
I20260928 23:58:45.337035 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:39567 every 8 connection(s)
I20260928 23:58:45.338091 6672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:45.342245 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6672
I20260928 23:58:45.342386 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:45.342728 20832 external_mini_cluster.cc:1428] Setting key f9e3356728494e1cd974918e7ecc354e
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
I20260928 23:58:45.355233 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.340302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:37567 (local address 127.20.88.62:35859)
0928 23:58:45.340453 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.340457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.341296 (+ 839us) server_negotiation.cc:407] Connection header received
0928 23:58:45.342855 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.342860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.342927 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.343032 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:45.344906 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.345792 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.347638 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.347893 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.348035 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.351186 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:45.351190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:45.351192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:45.351220 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:45.352820 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.353533 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.353536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.353538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.353621 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.353991 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.353994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.353996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.354203 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:45.354312 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.354904 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.355054 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:58:45.356194 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.340634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35859 (local address 127.20.88.2:37567)
0928 23:58:45.341125 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.341174 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.342546 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.343254 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.343265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.343980 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:45.344705 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.344722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.345964 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.345967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.347501 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.347516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.348089 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.348093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.348339 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.349028 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:45.349046 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:45.350975 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:45.353004 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.353012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.353036 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.353404 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.353716 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.353719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.353721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.353883 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.354312 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:45.354322 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:45.354631 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.354837 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.355214 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:58:45.357560 6801 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35859
I20260928 23:58:45.357856 6801 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:45.358506 6801 heartbeater.cc:511] Master 127.20.88.62:35859 requested a full tablet report, sending...
I20260928 23:58:45.359766 6476 ts_manager.cc:194] Registered new tserver with Master: 9814355b10014f0fbbab83abe4c92d64 (127.20.88.2:39567)
I20260928 23:58:45.360531 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:37567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:58:45.376029 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:45.492341 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:45.492622 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:45.492669 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:45.496872 6806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:45.496942 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:45.497006 6806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:45.502740 6806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:45.504020 6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:45.505125 6806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:58:45.516341 6806 server_base.cc:1034] running on GCE node
W20260928 23:58:45.516535 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.516584 6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.516607 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:45.517233 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:45.517952 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:45.519129 6806 hybrid_clock.cc:648] HybridClock initialized: now 1790639925519088 us; error 66 us; skew 500 ppm
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:45.522137 6806 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:58:45.523114 6806 webserver.cc:460] Webserver started at http://127.20.88.3:32781/ using document root <none> and password file <none>
I20260928 23:58:45.523725 6806 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:45.523818 6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:45.524055 6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:45.526352 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "f79d2515a23a45b999936d7e048d222b"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "8f8ad6b902f1087c1025c9d01089f193"
server_key_iv: "843d578eb3dd0e3546dedf29e03fda3d"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.526959 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "f79d2515a23a45b999936d7e048d222b"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "8f8ad6b902f1087c1025c9d01089f193"
server_key_iv: "843d578eb3dd0e3546dedf29e03fda3d"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.531031 6806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:58:45.533814 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.535198 6806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:45.535355 6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "f79d2515a23a45b999936d7e048d222b"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "8f8ad6b902f1087c1025c9d01089f193"
server_key_iv: "843d578eb3dd0e3546dedf29e03fda3d"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.535487 6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:45.558090 6806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:45.561830 6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:45.562076 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:45.562758 6806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:45.563807 6806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:45.563862 6806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.563915 6806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:45.563931 6806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:45.573268 6806 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:33021
I20260928 23:58:45.573295 6934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:33021 every 8 connection(s)
I20260928 23:58:45.574276 6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:45.583360 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6806
I20260928 23:58:45.583554 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:45.583913 20832 external_mini_cluster.cc:1428] Setting key a5a0fc9328db22563a0fe3fa3aa3dbb9
Sep 28 23:58:45 dist-test-slave-wvpw krb5kdc[6431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:45.589721 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.576359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:59623 (local address 127.20.88.62:35859)
0928 23:58:45.576529 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.576533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.577534 (+ 1001us) server_negotiation.cc:407] Connection header received
0928 23:58:45.578985 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.578987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.579038 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.579118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:45.580506 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.581050 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.582516 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.582743 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.582834 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.585918 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:45.585921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:45.585924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:45.585951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:45.587466 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.588045 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.588048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.588049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.588129 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.588526 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.588530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.588532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.588738 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:45.588850 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.589412 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.589555 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:58:45.590633 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.576823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:35859 (local address 127.20.88.3:59623)
0928 23:58:45.577351 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.577405 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.578744 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.579308 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.579315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.579775 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:45.580350 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.580367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.581194 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.581198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.582372 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.582384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.582898 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.582901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.583059 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.583788 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:45.583812 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:45.585719 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:45.587599 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.587606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.587625 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.587923 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.588221 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:45.588224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:45.588226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:45.588420 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:45.588852 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:45.588862 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:45.589152 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.589353 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.589731 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260928 23:58:45.591893 6935 heartbeater.cc:347] Connected to a master server at 127.20.88.62:35859
I20260928 23:58:45.592309 6935 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:45.592907 6935 heartbeater.cc:511] Master 127.20.88.62:35859 requested a full tablet report, sending...
I20260928 23:58:45.594091 6476 ts_manager.cc:194] Registered new tserver with Master: f79d2515a23a45b999936d7e048d222b (127.20.88.3:33021)
I20260928 23:58:45.594715 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:59623
I20260928 23:58:45.598232 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:45.610028 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.600319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.20.88.62:35859)
0928 23:58:45.600460 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.600465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.600608 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:58:45.600699 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.600701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.600745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.600813 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:45.601565 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.602119 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.603059 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.603308 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.603406 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.605232 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:45.605236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:45.605238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:45.605268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:45.607872 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.608590 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.608594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.608596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.608693 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:45.609059 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:45.609062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:45.609064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:45.609303 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:45.609476 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.609759 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.609863 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:58:45.613968 6476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:45.617632 6476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:45.633487 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.627308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:34583 (local address 127.0.0.1:60018)
0928 23:58:45.628353 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.628367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.628459 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.629088 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.629092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.629119 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:45.629336 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.629343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630828 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.630852 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.631948 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.631961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.632891 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.633013 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633051 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.633177 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.633323 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":52,"spinlock_wait_cycles":4480,"thread_start_us":90,"threads_started":1}
I20260928 23:58:45.633497 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.627704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.20.88.1:34583)
0928 23:58:45.628162 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.628167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.628554 (+ 387us) server_negotiation.cc:407] Connection header received
0928 23:58:45.628624 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.628628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.628787 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.628934 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:45.629466 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630639 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632085 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.632757 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632890 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633060 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.633208 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.633323 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":96,"threads_started":1}
I20260928 23:58:45.633713 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.626897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:39567 (local address 127.0.0.1:40996)
0928 23:58:45.627667 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.627727 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.627868 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.629088 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.629092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.629119 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:45.629336 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.629343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630365 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.630372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.631748 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.631758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.633153 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.633156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.633316 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633364 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.633431 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.633530 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":126,"spinlock_wait_cycles":10368,"thread_start_us":100,"threads_started":1}
I20260928 23:58:45.634207 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.627080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:33021 (local address 127.0.0.1:47246)
0928 23:58:45.627667 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.627727 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.627865 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.628952 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.628961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.628987 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:45.629154 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.629160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630775 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.630778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.631897 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.631906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.633153 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.633156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.633312 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633354 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.633431 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.634083 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":585,"thread_start_us":104,"threads_started":1}
I20260928 23:58:45.634255 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.627060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.20.88.2:39567)
0928 23:58:45.628470 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.628484 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.628510 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:58:45.628593 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.628598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.628787 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.628934 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:45.629485 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630229 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.631905 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.632821 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632946 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633086 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.634004 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.634104 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":130,"threads_started":1}
I20260928 23:58:45.634632 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.627711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.20.88.3:33021)
0928 23:58:45.628400 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.628413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.628438 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:58:45.628516 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.628523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.628754 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.629284 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:45.629443 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.630643 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632024 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.632848 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.632946 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.633086 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.633531 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.634485 (+ 954us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":159,"threads_started":1}
I20260928 23:58:45.637051 6735 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 (DEFAULT_TABLE table=test-table [id=3ba8947a214b43889224b444a5a6ba60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:45.637077 6601 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 (DEFAULT_TABLE table=test-table [id=3ba8947a214b43889224b444a5a6ba60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:45.638134 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:45.638146 6601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:45.639146 6869 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 (DEFAULT_TABLE table=test-table [id=3ba8947a214b43889224b444a5a6ba60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:45.640377 6869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:45.644706 6952 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Bootstrap starting.
I20260928 23:58:45.646405 6953 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: Bootstrap starting.
I20260928 23:58:45.646932 6952 tablet_bootstrap.cc:654] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:45.648089 6952 log.cc:824] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:45.648578 6954 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Bootstrap starting.
I20260928 23:58:45.648767 6953 tablet_bootstrap.cc:654] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:45.649644 6953 log.cc:824] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:45.650288 6952 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: No bootstrap required, opened a new log
I20260928 23:58:45.650507 6952 ts_tablet_manager.cc:1397] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:58:45.651055 6954 tablet_bootstrap.cc:654] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:45.651597 6953 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: No bootstrap required, opened a new log
I20260928 23:58:45.651803 6953 ts_tablet_manager.cc:1397] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:45.652359 6954 log.cc:824] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:45.654527 6954 tablet_bootstrap.cc:492] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: No bootstrap required, opened a new log
I20260928 23:58:45.654738 6954 ts_tablet_manager.cc:1397] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:45.654663 6952 raft_consensus.cc:348] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.654981 6952 raft_consensus.cc:374] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:45.655038 6952 raft_consensus.cc:729] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9814355b10014f0fbbab83abe4c92d64, State: Initialized, Role: FOLLOWER
I20260928 23:58:45.655565 6952 consensus_queue.cc:260] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.655723 6953 raft_consensus.cc:348] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.656008 6953 raft_consensus.cc:374] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:45.656060 6953 raft_consensus.cc:729] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb94e447c84ad2825121076c465e85, State: Initialized, Role: FOLLOWER
I20260928 23:58:45.656564 6953 consensus_queue.cc:260] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.656718 6952 ts_tablet_manager.cc:1428] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:45.657092 6801 heartbeater.cc:503] Master 127.20.88.62:35859 was elected leader, sending a full tablet report...
I20260928 23:58:45.657226 6667 heartbeater.cc:503] Master 127.20.88.62:35859 was elected leader, sending a full tablet report...
I20260928 23:58:45.657584 6953 ts_tablet_manager.cc:1428] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:45.658653 6954 raft_consensus.cc:348] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.658967 6954 raft_consensus.cc:374] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:45.659034 6954 raft_consensus.cc:729] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79d2515a23a45b999936d7e048d222b, State: Initialized, Role: FOLLOWER
I20260928 23:58:45.659502 6954 consensus_queue.cc:260] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.660391 6935 heartbeater.cc:503] Master 127.20.88.62:35859 was elected leader, sending a full tablet report...
I20260928 23:58:45.661677 6954 ts_tablet_manager.cc:1428] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:58:45.709057 6959 raft_consensus.cc:482] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:45.709283 6959 raft_consensus.cc:504] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.710537 6959 leader_election.cc:288] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9814355b10014f0fbbab83abe4c92d64 (127.20.88.2:39567), f79d2515a23a45b999936d7e048d222b (127.20.88.3:33021)
I20260928 23:58:45.715548 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.710917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:34771 (local address 127.20.88.2:39567)
0928 23:58:45.711121 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.711128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.711387 (+ 259us) server_negotiation.cc:407] Connection header received
0928 23:58:45.711743 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.711747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.711806 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.711854 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:45.712390 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.712972 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.714467 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.714891 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.714982 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.715080 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.715357 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.715443 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 23:58:45.715550 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.710960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:40641 (local address 127.20.88.3:33021)
0928 23:58:45.711224 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.711229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.711247 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:58:45.711503 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.711506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.711584 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.711674 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:45.712082 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.713249 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.714471 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.714924 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.715058 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.715128 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.715319 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.715408 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260928 23:58:45.715574 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.710783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:33021 (local address 127.20.88.1:40641)
0928 23:58:45.711161 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.711178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.711296 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.711799 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.711802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.711822 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:45.711949 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.711954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.713390 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.713394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.714298 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.714308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.715075 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.715079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.715175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.715210 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.715273 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.715388 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":45,"thread_start_us":185,"threads_started":1}
I20260928 23:58:45.715587 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.710790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:39567 (local address 127.20.88.1:34771)
0928 23:58:45.711294 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.711308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:45.711600 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:45.712045 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:45.712048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:45.712065 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:45.712184 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.712194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.713136 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.713140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.714297 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:45.714307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.715040 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.715044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:45.715158 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.715221 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 23:58:45.715323 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:58:45.715415 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":35,"thread_start_us":146,"threads_started":1}
I20260928 23:58:45.716432 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1cc21fd9f54d339c16c6a0759f1dc2" candidate_uuid: "0efb94e447c84ad2825121076c465e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d2515a23a45b999936d7e048d222b" is_pre_election: true
I20260928 23:58:45.716492 6755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1cc21fd9f54d339c16c6a0759f1dc2" candidate_uuid: "0efb94e447c84ad2825121076c465e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9814355b10014f0fbbab83abe4c92d64" is_pre_election: true
I20260928 23:58:45.716775 6889 raft_consensus.cc:2454] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0efb94e447c84ad2825121076c465e85 in term 0.
I20260928 23:58:45.716813 6755 raft_consensus.cc:2454] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0efb94e447c84ad2825121076c465e85 in term 0.
I20260928 23:58:45.717355 6557 leader_election.cc:302] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb94e447c84ad2825121076c465e85, f79d2515a23a45b999936d7e048d222b; no voters:
I20260928 23:58:45.717749 6959 raft_consensus.cc:2781] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:45.717859 6959 raft_consensus.cc:482] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:45.717927 6959 raft_consensus.cc:3037] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:45.719502 6959 raft_consensus.cc:504] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.719931 6959 leader_election.cc:288] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [CANDIDATE]: Term 1 election: Requested vote from peers 9814355b10014f0fbbab83abe4c92d64 (127.20.88.2:39567), f79d2515a23a45b999936d7e048d222b (127.20.88.3:33021)
I20260928 23:58:45.720376 6755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1cc21fd9f54d339c16c6a0759f1dc2" candidate_uuid: "0efb94e447c84ad2825121076c465e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9814355b10014f0fbbab83abe4c92d64"
I20260928 23:58:45.720413 6889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1cc21fd9f54d339c16c6a0759f1dc2" candidate_uuid: "0efb94e447c84ad2825121076c465e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d2515a23a45b999936d7e048d222b"
I20260928 23:58:45.720549 6755 raft_consensus.cc:3037] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:45.720577 6889 raft_consensus.cc:3037] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:45.721704 6889 raft_consensus.cc:2454] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0efb94e447c84ad2825121076c465e85 in term 1.
I20260928 23:58:45.721704 6755 raft_consensus.cc:2454] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0efb94e447c84ad2825121076c465e85 in term 1.
I20260928 23:58:45.722114 6557 leader_election.cc:302] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0efb94e447c84ad2825121076c465e85, 9814355b10014f0fbbab83abe4c92d64, f79d2515a23a45b999936d7e048d222b; no voters:
I20260928 23:58:45.722323 6959 raft_consensus.cc:2781] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:45.722591 6959 raft_consensus.cc:686] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [term 1 LEADER]: Becoming Leader. State: Replica: 0efb94e447c84ad2825121076c465e85, State: Running, Role: LEADER
I20260928 23:58:45.722990 6959 consensus_queue.cc:237] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } }
I20260928 23:58:45.726565 6474 catalog_manager.cc:5495] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0efb94e447c84ad2825121076c465e85 (127.20.88.1). New cstate: current_term: 1 leader_uuid: "0efb94e447c84ad2825121076c465e85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0efb94e447c84ad2825121076c465e85" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 34583 } health_report { overall_health: HEALTHY } } }
I20260928 23:58:45.741218 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.737491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.20.88.1:34583)
0928 23:58:45.737613 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:45.737617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:45.737631 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:58:45.737701 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:45.737703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:45.737756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:45.737882 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:45.738093 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.738645 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.739609 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:45.739878 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:45.739970 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:45.740339 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:45.740796 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:45.740961 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:45.741057 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:58:45.741098 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:58:45.748143 6889 raft_consensus.cc:1261] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 1 FOLLOWER]: Refusing update from remote peer 0efb94e447c84ad2825121076c465e85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:45.748143 6755 raft_consensus.cc:1261] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 1 FOLLOWER]: Refusing update from remote peer 0efb94e447c84ad2825121076c465e85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:45.749018 6963 consensus_queue.cc:1035] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9814355b10014f0fbbab83abe4c92d64" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 39567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:45.749284 6959 consensus_queue.cc:1035] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 0efb94e447c84ad2825121076c465e85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d2515a23a45b999936d7e048d222b" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:45.758368 6968 mvcc.cc:205] Tried to move back new op lower bound from 7334461135855202304 to 7334461135764422656. Current Snapshot: MvccSnapshot[applied={T|T < 7334461135855202304}]
I20260928 23:58:45.770298 6474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:45.770502 6474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:45.773522 6474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8a5c3aa258d4e75baeb1a85b4e9c140: Sending DeleteTablet for 3 replicas of tablet 9a1cc21fd9f54d339c16c6a0759f1dc2
I20260928 23:58:45.774513 6735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:45 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:40996
I20260928 23:58:45.774518 6869 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:45 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:47246
I20260928 23:58:45.774516 6601 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:45 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:60018
I20260928 23:58:45.775125 6976 tablet_replica.cc:330] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: stopping tablet replica
I20260928 23:58:45.775286 6977 tablet_replica.cc:330] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: stopping tablet replica
I20260928 23:58:45.775537 6976 raft_consensus.cc:2229] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:45.775557 6977 raft_consensus.cc:2229] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:45.775578 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6538
I20260928 23:58:45.775902 6976 raft_consensus.cc:2258] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:45.777297 6977 raft_consensus.cc:2258] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:45.778779 6976 ts_tablet_manager.cc:1905] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:45.778777 6977 ts_tablet_manager.cc:1905] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:45.781692 6976 ts_tablet_manager.cc:1918] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:45.781692 6977 ts_tablet_manager.cc:1918] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:45.781813 6976 log.cc:1196] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/9a1cc21fd9f54d339c16c6a0759f1dc2
I20260928 23:58:45.781813 6977 log.cc:1196] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/9a1cc21fd9f54d339c16c6a0759f1dc2
I20260928 23:58:45.782162 6976 ts_tablet_manager.cc:1939] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P f79d2515a23a45b999936d7e048d222b: Deleting consensus metadata
I20260928 23:58:45.782163 6977 ts_tablet_manager.cc:1939] T 9a1cc21fd9f54d339c16c6a0759f1dc2 P 9814355b10014f0fbbab83abe4c92d64: Deleting consensus metadata
W20260928 23:58:45.783397 6462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:34583 (error 108)
I20260928 23:58:45.783552 6463 catalog_manager.cc:4841] TS f79d2515a23a45b999936d7e048d222b (127.20.88.3:33021): tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 (table test-table [id=3ba8947a214b43889224b444a5a6ba60]) successfully deleted
I20260928 23:58:45.783679 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6672
I20260928 23:58:45.784471 6460 catalog_manager.cc:4841] TS 9814355b10014f0fbbab83abe4c92d64 (127.20.88.2:39567): tablet 9a1cc21fd9f54d339c16c6a0759f1dc2 (table test-table [id=3ba8947a214b43889224b444a5a6ba60]) successfully deleted
I20260928 23:58:45.785466 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:45.785109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:34583 (local address 127.0.0.1:60050)
0928 23:58:45.785270 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:45.785357 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:34583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:58:45.785640 6462 catalog_manager.cc:4584] TS 0efb94e447c84ad2825121076c465e85 (127.20.88.1:34583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a1cc21fd9f54d339c16c6a0759f1dc2: Network error: Client connection negotiation failed: client connection to 127.20.88.1:34583: connect: Connection refused (error 111)
I20260928 23:58:45.790232 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6806
I20260928 23:58:45.796368 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6444
2026-09-28T23:58:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T23:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:45Z Disabled control of system clock
I20260928 23:58:45.828353 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37329
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:45.960913 6983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:45.961234 6983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:45.961273 6983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:45.965719 6983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:45.965801 6983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:45.965816 6983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:45.965835 6983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:45.965852 6983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:45.971005 6983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:45.972285 6983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:45.973763 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:45.982831 6988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:45.982934 6989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:45.983150 6983 server_base.cc:1034] running on GCE node
W20260928 23:58:45.982810 6991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:45.983824 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:45.985131 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:45.986332 6983 hybrid_clock.cc:648] HybridClock initialized: now 1790639925986310 us; error 39 us; skew 500 ppm
I20260928 23:58:45.988606 6983 webserver.cc:460] Webserver started at http://127.20.88.62:46559/ using document root <none> and password file <none>
I20260928 23:58:45.989228 6983 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:45.989322 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:45.989558 6983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:45.991715 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "7f90421a8a454b9888dd29c824fc3493"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "39734d1a38398b79a322fc45355bb5d1"
server_key_iv: "4ed720230234db16f3b292bbc6766678"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.992372 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "7f90421a8a454b9888dd29c824fc3493"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "39734d1a38398b79a322fc45355bb5d1"
server_key_iv: "4ed720230234db16f3b292bbc6766678"
server_key_version: "encryptionkey@0"
I20260928 23:58:45.996460 6983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:45.998987 6997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.000424 6983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:46.000576 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "7f90421a8a454b9888dd29c824fc3493"
format_stamp: "Formatted at 2026-09-28 23:58:45 on dist-test-slave-wvpw"
server_key: "39734d1a38398b79a322fc45355bb5d1"
server_key_iv: "4ed720230234db16f3b292bbc6766678"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.000703 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:46.016311 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:46.017161 6983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:46.017400 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:46.024919 6983 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:37329
I20260928 23:58:46.024947 7049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:37329 every 8 connection(s)
I20260928 23:58:46.025990 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:46.029410 7050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:46.034991 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 6983
I20260928 23:58:46.035154 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:46.035300 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: Bootstrap starting.
I20260928 23:58:46.035497 20832 external_mini_cluster.cc:1428] Setting key 135967301213a1538908d66f1f719ffb
I20260928 23:58:46.038509 7050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:46.039316 7050 log.cc:824] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:46.041272 7050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: No bootstrap required, opened a new log
I20260928 23:58:46.044865 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.036778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.20.88.62:37329)
0928 23:58:46.037475 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.037491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.037529 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:58:46.038218 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.038237 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.038678 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.039053 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:46.039537 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.040569 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.041497 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.041971 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.042144 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.043048 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:46.043053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:46.043066 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:46.043476 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:46.043532 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:46.043553 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:46.043659 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.043809 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.044075 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":110,"threads_started":1}
I20260928 23:58:46.045181 7050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } }
I20260928 23:58:46.045429 7050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:46.045516 7050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f90421a8a454b9888dd29c824fc3493, State: Initialized, Role: FOLLOWER
I20260928 23:58:46.046027 7050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } }
I20260928 23:58:46.046185 7050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:46.046267 7050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:46.046384 7050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:46.047456 7050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } }
I20260928 23:58:46.047811 7050 leader_election.cc:302] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f90421a8a454b9888dd29c824fc3493; no voters:
I20260928 23:58:46.048130 7050 leader_election.cc:288] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:46.048306 7055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:46.048554 7055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [term 1 LEADER]: Becoming Leader. State: Replica: 7f90421a8a454b9888dd29c824fc3493, State: Running, Role: LEADER
I20260928 23:58:46.048916 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } }
I20260928 23:58:46.049453 7050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:46.050721 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f90421a8a454b9888dd29c824fc3493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } } }
I20260928 23:58:46.050881 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:46.050721 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f90421a8a454b9888dd29c824fc3493. Latest consensus state: current_term: 1 leader_uuid: "7f90421a8a454b9888dd29c824fc3493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90421a8a454b9888dd29c824fc3493" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37329 } } }
I20260928 23:58:46.051041 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:46.051301 7064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:46.054832 7064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:46.061177 7064 catalog_manager.cc:1347] Generated new cluster ID: ee93c0530dfe404595e14e503284cf25
I20260928 23:58:46.061282 7064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:46.081727 7064 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:46.083178 7064 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:46.092967 7064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: Generated new TSK 0
I20260928 23:58:46.093950 7064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:46.108081 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--builtin_ntp_servers=127.20.88.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:46.232945 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:46.233271 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:46.233312 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:46.239114 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:46.239195 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:46.239295 7074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:46.244664 7074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:46.245899 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:46.247382 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:46.258703 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.258706 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.258773 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:46.258764 7074 server_base.cc:1034] running on GCE node
I20260928 23:58:46.259409 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:46.260192 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:46.261408 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790639926261390 us; error 36 us; skew 500 ppm
I20260928 23:58:46.263531 7074 webserver.cc:460] Webserver started at http://127.20.88.1:35885/ using document root <none> and password file <none>
I20260928 23:58:46.264214 7074 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:46.264366 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:46.264613 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:46.266909 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "b68cc57196d148acb0d0122e6de6764f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "e7e1956f1f4c9207f4dd9a0b15270c85"
server_key_iv: "c332cd4edba6285f6f1916f5fc3aa30f"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.267549 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "b68cc57196d148acb0d0122e6de6764f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "e7e1956f1f4c9207f4dd9a0b15270c85"
server_key_iv: "c332cd4edba6285f6f1916f5fc3aa30f"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.272069 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 23:58:46.275110 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.276636 7074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:46.276810 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "b68cc57196d148acb0d0122e6de6764f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "e7e1956f1f4c9207f4dd9a0b15270c85"
server_key_iv: "c332cd4edba6285f6f1916f5fc3aa30f"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.277006 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:46.299706 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:46.300567 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:46.300797 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:46.301450 7074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:46.302524 7074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:46.302603 7074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.302675 7074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:46.302724 7074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.312940 7074 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39913
I20260928 23:58:46.312950 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39913 every 8 connection(s)
I20260928 23:58:46.313985 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:46.315843 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7074
I20260928 23:58:46.316046 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:46.316421 20832 external_mini_cluster.cc:1428] Setting key cdcbbf453566b82ddef7b0213f0d26af
I20260928 23:58:46.318480 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--builtin_ntp_servers=127.20.88.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:46.325687 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.316109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38973 (local address 127.20.88.62:37329)
0928 23:58:46.316297 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.316303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.317615 (+ 1312us) server_negotiation.cc:407] Connection header received
0928 23:58:46.319281 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.319285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.319347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.319440 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:46.320906 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.321682 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.323022 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.323467 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.323608 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.324994 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:46.325002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:46.325007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:46.325031 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:46.325061 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:46.325077 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:46.325168 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.325346 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.325439 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:58:46.326700 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.316605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37329 (local address 127.20.88.1:38973)
0928 23:58:46.317438 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.317498 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.319032 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.319656 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.319666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.319805 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:46.320674 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.320700 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.321847 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.321850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.322868 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.322880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.323660 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.323663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.323837 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.324669 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:46.324671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:46.324722 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:46.324808 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:46.325184 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:46.325192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:46.325197 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.325299 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.325582 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":148,"threads_started":1}
I20260928 23:58:46.328169 7202 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37329
I20260928 23:58:46.328470 7202 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:46.329051 7202 heartbeater.cc:511] Master 127.20.88.62:37329 requested a full tablet report, sending...
I20260928 23:58:46.330792 7014 ts_manager.cc:194] Registered new tserver with Master: b68cc57196d148acb0d0122e6de6764f (127.20.88.1:39913)
I20260928 23:58:46.332495 7014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.1:38973
W20260928 23:58:46.439172 7205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:46.439602 7205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:46.439672 7205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:46.444027 7205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:46.444118 7205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:46.444177 7205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:46.449497 7205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:46.450861 7205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:46.452654 7205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:46.463047 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.463042 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.463047 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:46.463295 7205 server_base.cc:1034] running on GCE node
I20260928 23:58:46.463835 7205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:46.464582 7205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:46.465821 7205 hybrid_clock.cc:648] HybridClock initialized: now 1790639926465767 us; error 36 us; skew 500 ppm
I20260928 23:58:46.467835 7205 webserver.cc:460] Webserver started at http://127.20.88.2:35157/ using document root <none> and password file <none>
I20260928 23:58:46.468552 7205 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:46.468652 7205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:46.468894 7205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:46.471110 7205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "206330ab5f5c479b9dfdc4133c3276f4"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "7a8c687346739019c4ffde84f61fde0d"
server_key_iv: "0ae64cbc3554926fe97bf88986bc4d9c"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.471701 7205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "206330ab5f5c479b9dfdc4133c3276f4"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "7a8c687346739019c4ffde84f61fde0d"
server_key_iv: "0ae64cbc3554926fe97bf88986bc4d9c"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.475713 7205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:46.478425 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.479801 7205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:58:46.479976 7205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "206330ab5f5c479b9dfdc4133c3276f4"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "7a8c687346739019c4ffde84f61fde0d"
server_key_iv: "0ae64cbc3554926fe97bf88986bc4d9c"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.480140 7205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:46.505172 7205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:46.505975 7205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:46.506188 7205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:46.506815 7205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:46.507822 7205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:46.507891 7205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.507967 7205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:46.508008 7205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.517818 7205 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:36083
I20260928 23:58:46.517843 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:36083 every 8 connection(s)
I20260928 23:58:46.518824 7205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:46.526754 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7205
I20260928 23:58:46.526962 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:46.527329 20832 external_mini_cluster.cc:1428] Setting key 50a642596c59ba33eed5f4aedc35f427
I20260928 23:58:46.529434 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.520966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:44795 (local address 127.20.88.62:37329)
0928 23:58:46.521139 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.521144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.522276 (+ 1132us) server_negotiation.cc:407] Connection header received
0928 23:58:46.523684 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.523687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.523744 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.523857 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:46.524913 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.525478 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.526968 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.527245 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.527377 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.528835 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:46.528842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:46.528848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:46.528865 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:46.528892 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:46.528904 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:46.528982 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.529189 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.529258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:58:46.529793 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--builtin_ntp_servers=127.20.88.20:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:46.530414 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.521420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37329 (local address 127.20.88.2:44795)
0928 23:58:46.522122 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.522164 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.523454 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.524033 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.524040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.524127 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:46.524747 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.524761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.525618 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.525621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.526794 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.526808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.527485 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.527495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.527654 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.528482 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:46.528484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:46.528546 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:46.528635 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:46.529002 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:46.529013 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:46.529020 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.529167 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.529430 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
I20260928 23:58:46.531785 7333 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37329
I20260928 23:58:46.532064 7333 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:46.532712 7333 heartbeater.cc:511] Master 127.20.88.62:37329 requested a full tablet report, sending...
I20260928 23:58:46.533928 7014 ts_manager.cc:194] Registered new tserver with Master: 206330ab5f5c479b9dfdc4133c3276f4 (127.20.88.2:36083)
I20260928 23:58:46.534888 7014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.2:44795
W20260928 23:58:46.651886 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:46.652210 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:46.652311 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:46.656885 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:46.656960 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:46.657022 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:46.662293 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:46.663446 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:46.664999 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:46.675647 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.675668 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:46.675652 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:46.675976 7336 server_base.cc:1034] running on GCE node
I20260928 23:58:46.676549 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:46.677250 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:46.678449 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790639926678424 us; error 37 us; skew 500 ppm
I20260928 23:58:46.680745 7336 webserver.cc:460] Webserver started at http://127.20.88.3:42589/ using document root <none> and password file <none>
I20260928 23:58:46.681511 7336 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:46.681591 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:46.681834 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:46.684718 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "36fd670180c049cab12f32fe36b6708f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "63c727f7ebc127169f1b28b8b7cc0074"
server_key_iv: "0d564d41d546e4bb9bffb1fbf6f16fee"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.685431 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "36fd670180c049cab12f32fe36b6708f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "63c727f7ebc127169f1b28b8b7cc0074"
server_key_iv: "0d564d41d546e4bb9bffb1fbf6f16fee"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.689707 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:46.692798 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.694537 7336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:46.694717 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "36fd670180c049cab12f32fe36b6708f"
format_stamp: "Formatted at 2026-09-28 23:58:46 on dist-test-slave-wvpw"
server_key: "63c727f7ebc127169f1b28b8b7cc0074"
server_key_iv: "0d564d41d546e4bb9bffb1fbf6f16fee"
server_key_version: "encryptionkey@0"
I20260928 23:58:46.694872 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:46.730984 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:46.731772 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:46.732005 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:46.732672 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:46.733673 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:46.733722 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.733763 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:46.733822 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:46.742771 7336 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:34151
I20260928 23:58:46.742789 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:34151 every 8 connection(s)
I20260928 23:58:46.743726 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:46.748073 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7336
I20260928 23:58:46.748220 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:46.748639 20832 external_mini_cluster.cc:1428] Setting key 49ed0dddc1eb0d3cb53102929de62a5e
I20260928 23:58:46.754763 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.745885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:33233 (local address 127.20.88.62:37329)
0928 23:58:46.746041 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.746045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.747151 (+ 1106us) server_negotiation.cc:407] Connection header received
0928 23:58:46.749056 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.749061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.749129 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.749241 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:46.750612 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.751363 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.752588 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.752883 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.752981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.754230 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:46.754234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:46.754236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:46.754258 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:46.754280 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:46.754288 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:46.754352 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.754548 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.754606 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:58:46.755705 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.746268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37329 (local address 127.20.88.3:33233)
0928 23:58:46.746996 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.747041 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.748720 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.749486 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.749500 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.749632 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:46.750445 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.750462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.751486 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.751489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.752464 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.752475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.753036 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.753039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.753195 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.753935 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:46.753936 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:46.753981 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:46.754057 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:46.754388 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:46.754397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:46.754402 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.754537 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.754832 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260928 23:58:46.756920 7464 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37329
I20260928 23:58:46.757182 7464 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:46.757731 7464 heartbeater.cc:511] Master 127.20.88.62:37329 requested a full tablet report, sending...
I20260928 23:58:46.758783 7014 ts_manager.cc:194] Registered new tserver with Master: 36fd670180c049cab12f32fe36b6708f (127.20.88.3:34151)
I20260928 23:58:46.759348 7014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.3:33233
I20260928 23:58:46.762933 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:46.769004 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.765000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.20.88.62:37329)
0928 23:58:46.765135 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.765139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.765338 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:58:46.765391 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.765393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.765435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.765504 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:46.765758 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.766295 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.767212 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.767492 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.767591 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.768539 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:46.768543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:46.768545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:46.768560 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:46.768582 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:46.768591 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:46.768713 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.768831 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.768874 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:58:46.772665 7014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:46.775861 7014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:46.791298 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.785382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.20.88.2:36083)
0928 23:58:46.785923 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.785934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.786773 (+ 839us) server_negotiation.cc:407] Connection header received
0928 23:58:46.786979 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.786985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.787138 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.787287 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:46.787490 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788323 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.789959 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.790581 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.790669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.790884 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.791008 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.791121 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":105,"threads_started":1}
I20260928 23:58:46.791302 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.785163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34151 (local address 127.0.0.1:53000)
0928 23:58:46.785997 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.786054 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.786285 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.787058 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.787064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.787101 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:46.787313 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.787322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788807 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.788810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.789997 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.790008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.790787 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.790791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.790874 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.790901 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.790967 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.791143 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":131,"spinlock_wait_cycles":7424,"thread_start_us":93,"threads_started":1}
I20260928 23:58:46.791298 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.785251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.20.88.3:34151)
0928 23:58:46.785566 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.785579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.786153 (+ 574us) server_negotiation.cc:407] Connection header received
0928 23:58:46.786482 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.786489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.786701 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.786839 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:46.787453 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788669 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.790120 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.790660 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.790770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.790975 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.791025 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.791123 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":117,"threads_started":1}
I20260928 23:58:46.791294 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.785163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36083 (local address 127.0.0.1:49260)
0928 23:58:46.786162 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.786175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.786837 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.787266 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.787269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.787288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:46.787395 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.787405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788439 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.788442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.789828 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.789839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.790709 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.790715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.790826 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.790864 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.790976 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.791142 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":457,"thread_start_us":92,"threads_started":1}
I20260928 23:58:46.794996 7398 tablet_service.cc:1467] Processing CreateTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 (DEFAULT_TABLE table=test-table [id=8c2b3fd4df1541879faea713a60f6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:46.794996 7267 tablet_service.cc:1467] Processing CreateTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 (DEFAULT_TABLE table=test-table [id=8c2b3fd4df1541879faea713a60f6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:46.796182 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b67328d5754b92b4997a1fd61aa1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:46.796182 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b67328d5754b92b4997a1fd61aa1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:46.799479 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.785949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39913 (local address 127.0.0.1:55822)
0928 23:58:46.786422 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:46.786435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:46.786509 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:46.787058 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:46.787064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:46.787099 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:46.787322 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.787328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788784 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.788788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.790017 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:46.790023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.798925 (+ 8902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.798928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:46.799018 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.799044 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:58:46.799111 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:58:46.799184 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":123,"spinlock_wait_cycles":4864,"thread_start_us":92,"threads_started":1}
I20260928 23:58:46.800154 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:46.786115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.20.88.1:39913)
0928 23:58:46.786494 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:46.786500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:46.786523 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:58:46.786615 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:46.786619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:46.786765 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:46.786911 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:46.787453 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.788669 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.791567 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:46.792407 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:46.792511 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:46.799441 (+ 6930us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:46.799516 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:58:46.800012 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":79,"threads_started":1}
I20260928 23:58:46.803081 7136 tablet_service.cc:1467] Processing CreateTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 (DEFAULT_TABLE table=test-table [id=8c2b3fd4df1541879faea713a60f6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:46.804258 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b67328d5754b92b4997a1fd61aa1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:46.804291 7481 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Bootstrap starting.
I20260928 23:58:46.805747 7482 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Bootstrap starting.
I20260928 23:58:46.806846 7481 tablet_bootstrap.cc:654] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:46.807698 7482 tablet_bootstrap.cc:654] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:46.807786 7481 log.cc:824] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:46.808557 7482 log.cc:824] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:46.810036 7481 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: No bootstrap required, opened a new log
I20260928 23:58:46.810245 7482 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: No bootstrap required, opened a new log
I20260928 23:58:46.810247 7481 ts_tablet_manager.cc:1397] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:46.810451 7482 ts_tablet_manager.cc:1397] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:58:46.812350 7485 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Bootstrap starting.
I20260928 23:58:46.814432 7482 raft_consensus.cc:348] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.814432 7481 raft_consensus.cc:348] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.814759 7482 raft_consensus.cc:374] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:46.814759 7481 raft_consensus.cc:374] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:46.814818 7481 raft_consensus.cc:729] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fd670180c049cab12f32fe36b6708f, State: Initialized, Role: FOLLOWER
I20260928 23:58:46.814831 7482 raft_consensus.cc:729] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206330ab5f5c479b9dfdc4133c3276f4, State: Initialized, Role: FOLLOWER
I20260928 23:58:46.815342 7485 tablet_bootstrap.cc:654] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:46.815325 7481 consensus_queue.cc:260] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.815408 7482 consensus_queue.cc:260] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.816037 7464 heartbeater.cc:503] Master 127.20.88.62:37329 was elected leader, sending a full tablet report...
I20260928 23:58:46.816290 7333 heartbeater.cc:503] Master 127.20.88.62:37329 was elected leader, sending a full tablet report...
I20260928 23:58:46.816488 7481 ts_tablet_manager.cc:1428] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:58:46.816638 7485 log.cc:824] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:46.817073 7482 ts_tablet_manager.cc:1428] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:58:46.819306 7485 tablet_bootstrap.cc:492] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: No bootstrap required, opened a new log
I20260928 23:58:46.819533 7485 ts_tablet_manager.cc:1397] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:58:46.823127 7485 raft_consensus.cc:348] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.823433 7485 raft_consensus.cc:374] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:46.823516 7485 raft_consensus.cc:729] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68cc57196d148acb0d0122e6de6764f, State: Initialized, Role: FOLLOWER
I20260928 23:58:46.824005 7485 consensus_queue.cc:260] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:46.824741 7202 heartbeater.cc:503] Master 127.20.88.62:37329 was elected leader, sending a full tablet report...
I20260928 23:58:46.825011 7485 ts_tablet_manager.cc:1428] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Time spent starting tablet: real 0.005s user 0.004s sys 0.003s
W20260928 23:58:46.994771 7465 tablet.cc:2369] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:58:47.020188 7334 tablet.cc:2369] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:58:47.065410 7203 tablet.cc:2369] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:47.095517 7487 raft_consensus.cc:482] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:47.095788 7487 raft_consensus.cc:504] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:47.097087 7487 leader_election.cc:288] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fd670180c049cab12f32fe36b6708f (127.20.88.3:34151), b68cc57196d148acb0d0122e6de6764f (127.20.88.1:39913)
I20260928 23:58:47.101603 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.097475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:36059 (local address 127.20.88.3:34151)
0928 23:58:47.097613 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:47.097618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:47.097970 (+ 352us) server_negotiation.cc:407] Connection header received
0928 23:58:47.098112 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:47.098115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:47.098155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:47.098249 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:47.098562 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.099216 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.100462 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.100965 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.101050 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:47.101110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:47.101375 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:58:47.101459 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:58:47.101603 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.097388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:34151 (local address 127.20.88.2:36059)
0928 23:58:47.097876 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:47.097888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:47.098020 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:47.098294 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:47.098298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:47.098318 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:47.098426 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:47.098430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.099331 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.099337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:47.100346 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:47.100356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.101112 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.101116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:47.101228 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:47.101260 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:58:47.101344 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:58:47.101426 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":102,"thread_start_us":68,"threads_started":1}
I20260928 23:58:47.101787 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.097385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39913 (local address 127.20.88.2:37189)
0928 23:58:47.097752 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:47.097765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:47.097906 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:47.098243 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:47.098246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:47.098261 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:47.098401 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:47.098406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.099338 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.099343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:47.100512 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:47.100523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.101309 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.101311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:47.101432 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:47.101454 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:58:47.101533 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:58:47.101599 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
I20260928 23:58:47.101817 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.097487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:37189 (local address 127.20.88.1:39913)
0928 23:58:47.097617 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:47.097622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:47.097854 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:58:47.098072 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:47.098079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:47.098143 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:47.098201 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:47.098515 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.099216 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.100658 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.101164 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.101275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:47.101350 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:47.101566 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 23:58:47.101658 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:58:47.102537 7156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b67328d5754b92b4997a1fd61aa1c6" candidate_uuid: "206330ab5f5c479b9dfdc4133c3276f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68cc57196d148acb0d0122e6de6764f" is_pre_election: true
I20260928 23:58:47.102584 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b67328d5754b92b4997a1fd61aa1c6" candidate_uuid: "206330ab5f5c479b9dfdc4133c3276f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fd670180c049cab12f32fe36b6708f" is_pre_election: true
I20260928 23:58:47.102875 7156 raft_consensus.cc:2454] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206330ab5f5c479b9dfdc4133c3276f4 in term 0.
I20260928 23:58:47.102875 7418 raft_consensus.cc:2454] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206330ab5f5c479b9dfdc4133c3276f4 in term 0.
I20260928 23:58:47.103461 7220 leader_election.cc:302] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 206330ab5f5c479b9dfdc4133c3276f4, 36fd670180c049cab12f32fe36b6708f, b68cc57196d148acb0d0122e6de6764f; no voters:
I20260928 23:58:47.103778 7487 raft_consensus.cc:2781] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:47.103888 7487 raft_consensus.cc:482] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:47.103951 7487 raft_consensus.cc:3037] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:47.105067 7487 raft_consensus.cc:504] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:47.105477 7487 leader_election.cc:288] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [CANDIDATE]: Term 1 election: Requested vote from peers 36fd670180c049cab12f32fe36b6708f (127.20.88.3:34151), b68cc57196d148acb0d0122e6de6764f (127.20.88.1:39913)
I20260928 23:58:47.106035 7156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b67328d5754b92b4997a1fd61aa1c6" candidate_uuid: "206330ab5f5c479b9dfdc4133c3276f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68cc57196d148acb0d0122e6de6764f"
I20260928 23:58:47.106035 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b67328d5754b92b4997a1fd61aa1c6" candidate_uuid: "206330ab5f5c479b9dfdc4133c3276f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fd670180c049cab12f32fe36b6708f"
I20260928 23:58:47.106181 7418 raft_consensus.cc:3037] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:47.106181 7156 raft_consensus.cc:3037] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:47.107715 7156 raft_consensus.cc:2454] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206330ab5f5c479b9dfdc4133c3276f4 in term 1.
I20260928 23:58:47.107715 7418 raft_consensus.cc:2454] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206330ab5f5c479b9dfdc4133c3276f4 in term 1.
I20260928 23:58:47.108132 7220 leader_election.cc:302] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206330ab5f5c479b9dfdc4133c3276f4, b68cc57196d148acb0d0122e6de6764f; no voters:
I20260928 23:58:47.108397 7487 raft_consensus.cc:2781] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:47.108618 7487 raft_consensus.cc:686] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 1 LEADER]: Becoming Leader. State: Replica: 206330ab5f5c479b9dfdc4133c3276f4, State: Running, Role: LEADER
I20260928 23:58:47.108990 7487 consensus_queue.cc:237] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } }
I20260928 23:58:47.112480 7014 catalog_manager.cc:5495] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 206330ab5f5c479b9dfdc4133c3276f4 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "206330ab5f5c479b9dfdc4133c3276f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206330ab5f5c479b9dfdc4133c3276f4" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 36083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 } health_report { overall_health: UNKNOWN } } }
I20260928 23:58:47.160144 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.156369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.20.88.2:36083)
0928 23:58:47.156508 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:47.156512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:47.156526 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:58:47.156651 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:47.156653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:47.156710 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:47.156818 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:47.157081 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.157607 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.158446 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:47.158656 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:47.158748 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:47.159156 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:47.159589 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:47.159709 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:47.159898 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:58:47.159976 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:58:47.167135 7418 raft_consensus.cc:1261] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 1 FOLLOWER]: Refusing update from remote peer 206330ab5f5c479b9dfdc4133c3276f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:47.167135 7156 raft_consensus.cc:1261] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 1 FOLLOWER]: Refusing update from remote peer 206330ab5f5c479b9dfdc4133c3276f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:47.168164 7492 consensus_queue.cc:1035] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68cc57196d148acb0d0122e6de6764f" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:47.168478 7487 consensus_queue.cc:1035] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fd670180c049cab12f32fe36b6708f" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 34151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:47.187794 7013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:47.188087 7013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:47.191134 7013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f90421a8a454b9888dd29c824fc3493: Sending DeleteTablet for 3 replicas of tablet d0b67328d5754b92b4997a1fd61aa1c6
I20260928 23:58:47.192221 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:47 UTC) from {username='slave'} at 127.0.0.1:49260
I20260928 23:58:47.192220 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:47 UTC) from {username='slave'} at 127.0.0.1:53000
I20260928 23:58:47.192382 7136 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b67328d5754b92b4997a1fd61aa1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:47 UTC) from {username='slave'} at 127.0.0.1:55822
I20260928 23:58:47.192978 7505 tablet_replica.cc:330] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: stopping tablet replica
I20260928 23:58:47.193171 7507 tablet_replica.cc:330] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f: stopping tablet replica
I20260928 23:58:47.193463 7505 raft_consensus.cc:2229] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:47.193513 7507 raft_consensus.cc:2229] T d0b67328d5754b92b4997a1fd61aa1c6 P b68cc57196d148acb0d0122e6de6764f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:47.193718 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7074
I20260928 23:58:47.195099 7506 tablet_replica.cc:330] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: stopping tablet replica
I20260928 23:58:47.195497 7506 raft_consensus.cc:2229] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:47.195971 7506 raft_consensus.cc:2258] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:47.196574 7505 raft_consensus.cc:2258] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:47.197890 7506 ts_tablet_manager.cc:1905] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:47.198359 7505 ts_tablet_manager.cc:1905] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:47.201407 7505 ts_tablet_manager.cc:1918] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:47.201630 7505 log.cc:1196] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/d0b67328d5754b92b4997a1fd61aa1c6
I20260928 23:58:47.201746 7506 ts_tablet_manager.cc:1918] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:47.201864 7506 log.cc:1196] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/d0b67328d5754b92b4997a1fd61aa1c6
W20260928 23:58:47.201947 6998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:39913 (error 108)
I20260928 23:58:47.202211 7505 ts_tablet_manager.cc:1939] T d0b67328d5754b92b4997a1fd61aa1c6 P 36fd670180c049cab12f32fe36b6708f: Deleting consensus metadata
I20260928 23:58:47.202211 7506 ts_tablet_manager.cc:1939] T d0b67328d5754b92b4997a1fd61aa1c6 P 206330ab5f5c479b9dfdc4133c3276f4: Deleting consensus metadata
I20260928 23:58:47.203486 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7205
I20260928 23:58:47.203567 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.203213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39913 (local address 127.0.0.1:55828)
0928 23:58:47.203377 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:47.203468 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:58:47.203761 6998 catalog_manager.cc:4584] TS b68cc57196d148acb0d0122e6de6764f (127.20.88.1:39913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0b67328d5754b92b4997a1fd61aa1c6: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39913: connect: Connection refused (error 111)
I20260928 23:58:47.204090 6999 catalog_manager.cc:4841] TS 36fd670180c049cab12f32fe36b6708f (127.20.88.3:34151): tablet d0b67328d5754b92b4997a1fd61aa1c6 (table test-table [id=8c2b3fd4df1541879faea713a60f6309]) successfully deleted
I20260928 23:58:47.209878 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7336
I20260928 23:58:47.211119 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:47.210614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:36083 (local address 127.0.0.1:49272)
0928 23:58:47.210941 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:47.211006 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260928 23:58:47.215672 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 6983
2026-09-28T23:58:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1417 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:47 dist-test-slave-wvpw krb5kdc[7511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:47 dist-test-slave-wvpw krb5kdc[7511](info): set up 2 sockets
Sep 28 23:58:47 dist-test-slave-wvpw krb5kdc[7511](info): commencing operation
krb5kdc: starting...
W20260928 23:58:48.260833 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:48 dist-test-slave-wvpw krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:58:48.395920 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44491
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:48.515360 7524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:48.515693 7524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:48.515738 7524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:48.520005 7524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:48.520100 7524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:48.520120 7524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:48.520138 7524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:48.520153 7524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:48.526211 7524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:48.527369 7524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:48.528817 7524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:48.538333 7530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:48.538378 7529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:48.538365 7532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:48.538576 7524 server_base.cc:1034] running on GCE node
I20260928 23:58:48.539330 7524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:48.540584 7524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:48.541774 7524 hybrid_clock.cc:648] HybridClock initialized: now 1790639928541758 us; error 30 us; skew 500 ppm
Sep 28 23:58:48 dist-test-slave-wvpw krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639928, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:48.544937 7524 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:58:48.545957 7524 webserver.cc:460] Webserver started at http://127.20.88.62:37015/ using document root <none> and password file <none>
I20260928 23:58:48.546541 7524 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:48.546604 7524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:48.546821 7524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:48.549142 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "59abeb6ae5394409a3e1d8d9350ddd55"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "5b2184771deb0a1a4dfea921c0df1b9d"
server_key_iv: "d69a575775d648be4edd3f45270baa05"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.549710 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "59abeb6ae5394409a3e1d8d9350ddd55"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "5b2184771deb0a1a4dfea921c0df1b9d"
server_key_iv: "d69a575775d648be4edd3f45270baa05"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.553768 7524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:48.556582 7539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:48.557977 7524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:58:48.558138 7524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "59abeb6ae5394409a3e1d8d9350ddd55"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "5b2184771deb0a1a4dfea921c0df1b9d"
server_key_iv: "d69a575775d648be4edd3f45270baa05"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.558302 7524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:48.573529 7524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:48.576980 7524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:48.577201 7524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:48.584787 7524 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:44491
I20260928 23:58:48.584812 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:44491 every 8 connection(s)
I20260928 23:58:48.585954 7524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:48.589022 7592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:48.593231 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7524
I20260928 23:58:48.593400 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:48.593734 20832 external_mini_cluster.cc:1428] Setting key 710bae5d37c1203067d4830beaf531b7
I20260928 23:58:48.596123 7592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: Bootstrap starting.
I20260928 23:58:48.598670 7592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:48.599512 7592 log.cc:824] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:48.601857 7592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: No bootstrap required, opened a new log
Sep 28 23:58:48 dist-test-slave-wvpw krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:48.605384 7592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } }
I20260928 23:58:48.605756 7592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:48.605850 7592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59abeb6ae5394409a3e1d8d9350ddd55, State: Initialized, Role: FOLLOWER
I20260928 23:58:48.606410 7592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } }
I20260928 23:58:48.606547 7592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:48.606632 7592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:48.606771 7592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:48.607789 7592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } }
I20260928 23:58:48.608181 7592 leader_election.cc:302] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59abeb6ae5394409a3e1d8d9350ddd55; no voters:
I20260928 23:58:48.608628 7592 leader_election.cc:288] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:48.608721 7597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:48.608985 7597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [term 1 LEADER]: Becoming Leader. State: Replica: 59abeb6ae5394409a3e1d8d9350ddd55, State: Running, Role: LEADER
I20260928 23:58:48.609349 7597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } }
I20260928 23:58:48.610098 7592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:48.610461 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:48.595446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.20.88.62:44491)
0928 23:58:48.595980 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:48.595995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:48.596033 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:58:48.596815 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:48.596834 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:48.597197 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:48.597567 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:48.598426 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.599527 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.600700 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.601204 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.601396 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:48.604070 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:48.604076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:48.604092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:48.604124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:48.606727 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:48.607235 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:48.607242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:48.607248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:48.607383 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:48.607701 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:48.607704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:48.607706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:48.608220 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:48.608501 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:48.608938 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 23:58:48.609346 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":154,"threads_started":1}
I20260928 23:58:48.611011 7599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59abeb6ae5394409a3e1d8d9350ddd55. Latest consensus state: current_term: 1 leader_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } } }
I20260928 23:58:48.611164 7599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:48.611541 7598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abeb6ae5394409a3e1d8d9350ddd55" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44491 } } }
I20260928 23:58:48.611647 7598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55 [sys.catalog]: This master's current role is: LEADER
W20260928 23:58:48.615207 7612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:58:48.615293 7612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:58:48.615408 7606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:48.616299 7606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:48.622673 7606 catalog_manager.cc:1347] Generated new cluster ID: b6ec9afae6f84d429150de0dc3378517
I20260928 23:58:48.622794 7606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:48.634263 7606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:48.635417 7606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:48.647207 7606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: Generated new TSK 0
I20260928 23:58:48.648272 7606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.88.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
Entry for principal kudu/127.20.88.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab.
I20260928 23:58:48.692693 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:48.823292 7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:48.823614 7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:48.823660 7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:48.827996 7618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:48.828087 7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:48.828151 7618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:48.834517 7618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.1.keytab
--principal=kudu/127.20.88.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:48.835899 7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:48.837214 7618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:48.847810 7624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:48.847890 7618 server_base.cc:1034] running on GCE node
W20260928 23:58:48.847934 7626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:48.847915 7623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:48.848661 7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:48.849412 7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:48.850644 7618 hybrid_clock.cc:648] HybridClock initialized: now 1790639928850619 us; error 40 us; skew 500 ppm
Sep 28 23:58:48 dist-test-slave-wvpw krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639928, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:48.853998 7618 init.cc:377] Logged in from keytab as kudu/127.20.88.1@KRBTEST.COM (short username kudu)
I20260928 23:58:48.855123 7618 webserver.cc:460] Webserver started at http://127.20.88.1:35885/ using document root <none> and password file <none>
I20260928 23:58:48.855752 7618 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:48.855866 7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:48.856119 7618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:48.858431 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "d464a929631143c0a2d7aaff4a5bd44a"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "59db7b43bd3efa1245e0fc4125e25864"
server_key_iv: "a1da6bf2ea2f796d38434e41de548d2a"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.859097 7618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "d464a929631143c0a2d7aaff4a5bd44a"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "59db7b43bd3efa1245e0fc4125e25864"
server_key_iv: "a1da6bf2ea2f796d38434e41de548d2a"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.863701 7618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:48.866582 7633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:48.868187 7618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:58:48.868433 7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "d464a929631143c0a2d7aaff4a5bd44a"
format_stamp: "Formatted at 2026-09-28 23:58:48 on dist-test-slave-wvpw"
server_key: "59db7b43bd3efa1245e0fc4125e25864"
server_key_iv: "a1da6bf2ea2f796d38434e41de548d2a"
server_key_version: "encryptionkey@0"
I20260928 23:58:48.868574 7618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:48.890190 7618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:48.893854 7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:48.894086 7618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:48.894759 7618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:48.895675 7618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:48.895749 7618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:48.895815 7618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:48.895857 7618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:48.905987 7618 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:39963
I20260928 23:58:48.906009 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:39963 every 8 connection(s)
I20260928 23:58:48.906973 7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:48.911369 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7618
I20260928 23:58:48.911531 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:48.911870 20832 external_mini_cluster.cc:1428] Setting key 73f151699714d0386fcad66b0fc8724e
Sep 28 23:58:48 dist-test-slave-wvpw krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639928, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.1@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.2@KRBTEST.COM; defaulting to no policy
I20260928 23:58:48.924095 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:48.909055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:38229 (local address 127.20.88.62:44491)
0928 23:58:48.909229 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:48.909233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:48.910238 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 23:58:48.911945 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:48.911949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:48.912015 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:48.912108 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:48.913785 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.914335 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.916162 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.916630 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.916757 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:48.919745 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:48.919750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:48.919753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:48.919789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:48.921741 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:48.922343 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:48.922348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:48.922353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:48.922462 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:48.922864 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:48.922867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:48.922868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:48.923088 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:48.923191 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:48.923824 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 23:58:48.923958 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:58:48.925160 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:48.909494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44491 (local address 127.20.88.1:38229)
0928 23:58:48.910078 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:48.910125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:48.911704 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:48.912361 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:48.912370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:48.912897 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:48.913615 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:48.913632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.914498 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.914502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:48.915986 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:48.916002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:48.916812 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:48.916816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:48.916982 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:48.917650 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:48.917667 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:48.919511 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:48.921900 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:48.921906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:48.921923 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:48.922221 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:48.922579 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:48.922582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:48.922583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:48.922733 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:48.923240 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:48.923247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:48.923552 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 23:58:48.923755 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 23:58:48.924090 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":187,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.2@KRBTEST.COM" created.
I20260928 23:58:48.926796 7747 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44491
I20260928 23:58:48.927090 7747 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:48.927661 7747 heartbeater.cc:511] Master 127.20.88.62:44491 requested a full tablet report, sending...
I20260928 23:58:48.929483 7556 ts_manager.cc:194] Registered new tserver with Master: d464a929631143c0a2d7aaff4a5bd44a (127.20.88.1:39963)
I20260928 23:58:48.930691 7556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.1@KRBTEST.COM'} at 127.20.88.1:38229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
Entry for principal kudu/127.20.88.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab.
I20260928 23:58:48.944800 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:49.064167 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:49.064559 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:49.064637 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:49.069191 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:49.069274 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:49.069377 7752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:49.075459 7752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.2.keytab
--principal=kudu/127.20.88.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:49.076764 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:49.077884 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:49.088629 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:49.088819 7752 server_base.cc:1034] running on GCE node
W20260928 23:58:49.088613 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:49.088685 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:49.089342 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:49.090116 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:49.091295 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790639929091251 us; error 51 us; skew 500 ppm
Sep 28 23:58:49 dist-test-slave-wvpw krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639929, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:49.094584 7752 init.cc:377] Logged in from keytab as kudu/127.20.88.2@KRBTEST.COM (short username kudu)
I20260928 23:58:49.095552 7752 webserver.cc:460] Webserver started at http://127.20.88.2:34711/ using document root <none> and password file <none>
I20260928 23:58:49.096331 7752 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:49.096424 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:49.096617 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:49.098903 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "30f068b159a2478eb0f1197c04217552"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "36f5302f28e2653eb095d808f5137032"
server_key_iv: "5252d6882345d518fb6f2d323dab9042"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.099483 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "30f068b159a2478eb0f1197c04217552"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "36f5302f28e2653eb095d808f5137032"
server_key_iv: "5252d6882345d518fb6f2d323dab9042"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.103922 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 23:58:49.106855 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.108383 7752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:49.108539 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "30f068b159a2478eb0f1197c04217552"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "36f5302f28e2653eb095d808f5137032"
server_key_iv: "5252d6882345d518fb6f2d323dab9042"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.108678 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:49.133409 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:49.137142 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:49.137367 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:49.138010 7752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:49.138917 7752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:49.138988 7752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.139062 7752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:49.139106 7752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.149205 7752 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:35239
I20260928 23:58:49.149221 7880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:35239 every 8 connection(s)
I20260928 23:58:49.150246 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:49.152585 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7752
I20260928 23:58:49.152715 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:49.153023 20832 external_mini_cluster.cc:1428] Setting key 1cdf1a0502c84f149abff222df395a18
Sep 28 23:58:49 dist-test-slave-wvpw krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639929, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.2@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.88.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.3@KRBTEST.COM" created.
I20260928 23:58:49.166692 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.152445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:45765 (local address 127.20.88.62:44491)
0928 23:58:49.152590 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.152594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.153810 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 23:58:49.155387 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.155391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.155451 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.155530 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:49.156878 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.157436 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.158669 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.158923 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.159053 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.162213 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:49.162221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:49.162224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:49.162274 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:49.163877 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.164621 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.164625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.164626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.164712 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.165207 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.165210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.165212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.165508 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:49.165624 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.166232 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.166433 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:58:49.167567 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.152961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44491 (local address 127.20.88.2:45765)
0928 23:58:49.153575 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.153646 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.155114 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.155721 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.155729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.156124 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:49.156721 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.156737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.157588 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.157591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.158526 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.158536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.159105 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.159109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.159304 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.159976 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:49.159994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:49.161981 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:49.164005 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:49.164011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:49.164028 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:49.164421 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:49.164886 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:49.164889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:49.164890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:49.165064 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:49.165634 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:49.165641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:49.165940 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.166113 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.166485 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":155,"threads_started":1}
I20260928 23:58:49.169270 7881 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44491
I20260928 23:58:49.169560 7881 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:49.170218 7881 heartbeater.cc:511] Master 127.20.88.62:44491 requested a full tablet report, sending...
I20260928 23:58:49.171591 7556 ts_manager.cc:194] Registered new tserver with Master: 30f068b159a2478eb0f1197c04217552 (127.20.88.2:35239)
I20260928 23:58:49.172595 7556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.2@KRBTEST.COM'} at 127.20.88.2:45765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
Entry for principal kudu/127.20.88.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab.
I20260928 23:58:49.185714 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:49.306541 7886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:49.306849 7886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:49.306896 7886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:49.311508 7886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:49.311618 7886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:49.311703 7886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:49.317742 7886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.3.keytab
--principal=kudu/127.20.88.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:49.318841 7886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:49.319850 7886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:49.329809 7892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:49.329958 7886 server_base.cc:1034] running on GCE node
W20260928 23:58:49.330158 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:49.329828 7891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:49.330667 7886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:49.331346 7886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:49.332530 7886 hybrid_clock.cc:648] HybridClock initialized: now 1790639929332510 us; error 31 us; skew 500 ppm
Sep 28 23:58:49 dist-test-slave-wvpw krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639929, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:49.335343 7886 init.cc:377] Logged in from keytab as kudu/127.20.88.3@KRBTEST.COM (short username kudu)
I20260928 23:58:49.336305 7886 webserver.cc:460] Webserver started at http://127.20.88.3:33355/ using document root <none> and password file <none>
I20260928 23:58:49.336854 7886 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:49.336913 7886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:49.337086 7886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:49.339231 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "5a9a07d5791442fdbe4855f3d189fc50"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "3b12908d9872aa683be533bb5a8d73f5"
server_key_iv: "b66636c66dc005f7c06c20d19d3a8989"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.339814 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "5a9a07d5791442fdbe4855f3d189fc50"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "3b12908d9872aa683be533bb5a8d73f5"
server_key_iv: "b66636c66dc005f7c06c20d19d3a8989"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.343742 7886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:49.346388 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.347708 7886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:58:49.347965 7886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "5a9a07d5791442fdbe4855f3d189fc50"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "3b12908d9872aa683be533bb5a8d73f5"
server_key_iv: "b66636c66dc005f7c06c20d19d3a8989"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.348086 7886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:49.378288 7886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:49.381750 7886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:49.381976 7886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:49.382618 7886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:49.383509 7886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:49.383558 7886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.383632 7886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:49.383666 7886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.393232 7886 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:40513
I20260928 23:58:49.393299 8014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:40513 every 8 connection(s)
I20260928 23:58:49.394560 7886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:49.403452 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 7886
I20260928 23:58:49.403615 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:49.403934 20832 external_mini_cluster.cc:1428] Setting key 1138baa7b258804211cf199170a759df
Sep 28 23:58:49 dist-test-slave-wvpw krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639929, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.3@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:58:49.409664 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.396845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:32851 (local address 127.20.88.62:44491)
0928 23:58:49.397021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.397026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.398046 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 23:58:49.399426 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.399429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.399482 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.399566 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:49.400874 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.401546 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.402786 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.403013 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.403108 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.406219 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:49.406222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:49.406225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:49.406253 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:49.407527 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.408075 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.408079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.408081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.408164 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.408545 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.408548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.408550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.408748 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:49.408870 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.409373 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.409503 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 23:58:49.410593 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.397324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:44491 (local address 127.20.88.3:32851)
0928 23:58:49.397827 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.397905 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.399196 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.399742 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.399749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.400175 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:49.400707 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.400723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.401665 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.401667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.402608 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.402617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.403226 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.403230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.403458 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.404197 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:58:49.404219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:49.406022 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:49.407656 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:49.407662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:49.407680 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:49.407968 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:49.408283 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:58:49.408286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:58:49.408287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:58:49.408441 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:58:49.408857 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:49.408864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:49.409132 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.409318 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.409675 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260928 23:58:49.411995 8015 heartbeater.cc:347] Connected to a master server at 127.20.88.62:44491
I20260928 23:58:49.412278 8015 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:49.412793 8015 heartbeater.cc:511] Master 127.20.88.62:44491 requested a full tablet report, sending...
I20260928 23:58:49.413924 7556 ts_manager.cc:194] Registered new tserver with Master: 5a9a07d5791442fdbe4855f3d189fc50 (127.20.88.3:40513)
I20260928 23:58:49.414592 7556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.88.3@KRBTEST.COM'} at 127.20.88.3:32851
I20260928 23:58:49.418084 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:49.428323 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.420101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.20.88.62:44491)
0928 23:58:49.420280 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.420285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.420511 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:58:49.420610 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.420612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.420652 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.420720 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:49.421420 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.421931 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.422733 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.422969 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.423045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.424655 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:49.424659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:49.424661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:49.424696 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:49.426356 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.426879 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.426882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.426884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.426978 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:49.427287 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:49.427290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:49.427291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:49.427600 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:49.427722 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.427995 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.428108 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:58:49.431775 7556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:49.435269 7556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:49.451469 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.20.88.1:39963)
0928 23:58:49.446800 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.446805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.446824 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:58:49.446882 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.446886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.447032 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.447190 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:49.447508 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.448400 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.449899 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.450607 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.450722 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.450876 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.451155 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.451275 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":85,"threads_started":1}
I20260928 23:58:49.451462 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39963 (local address 127.0.0.1:53686)
0928 23:58:49.446562 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.446571 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.446623 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.447190 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.447196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.447219 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:49.447392 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.447398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.448509 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.448515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.449774 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.449790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.450750 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.450754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.450910 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.450982 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.451107 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.451268 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":20,"spinlock_wait_cycles":8832,"thread_start_us":85,"threads_started":1}
I20260928 23:58:49.451694 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:35239 (local address 127.0.0.1:55348)
0928 23:58:49.446218 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.446274 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.446399 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.447234 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.447238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.447270 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:49.447431 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.447439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.449168 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.449172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.450066 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.450076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.451270 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.451273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.451378 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.451401 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.451450 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.451540 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":29,"spinlock_wait_cycles":9088,"thread_start_us":123,"threads_started":1}
I20260928 23:58:49.452435 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.20.88.2:35239)
0928 23:58:49.446810 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.446822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.446843 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:58:49.446913 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.446918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.447060 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.447604 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:49.447823 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.449005 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.450275 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.451132 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.451976 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.452124 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.452173 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.452289 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":126,"threads_started":1}
I20260928 23:58:49.453469 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:40513 (local address 127.0.0.1:38826)
0928 23:58:49.446218 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.446277 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.446402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.447236 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.447238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.447272 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:49.447603 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.447609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.449168 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.449172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.451178 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.451193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.453028 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.453033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.453110 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.453129 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.453235 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.453318 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":238,"spinlock_wait_cycles":13056,"thread_start_us":91,"threads_started":1}
I20260928 23:58:49.453624 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.445219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.20.88.3:40513)
0928 23:58:49.446717 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.446731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.446759 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:58:49.446842 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.446848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.447060 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.447366 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:49.447737 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.448999 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.451328 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.452692 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.452809 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.452959 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.453267 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.453367 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1375,"thread_start_us":146,"threads_started":1}
I20260928 23:58:49.455552 7681 tablet_service.cc:1467] Processing CreateTablet for tablet ca5075b0905c4737ac82383705d710d5 (DEFAULT_TABLE table=test-table [id=fd381de0209a45bd9693945778bb22ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:49.455559 7815 tablet_service.cc:1467] Processing CreateTablet for tablet ca5075b0905c4737ac82383705d710d5 (DEFAULT_TABLE table=test-table [id=fd381de0209a45bd9693945778bb22ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:49.456804 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5075b0905c4737ac82383705d710d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:49.456805 7815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5075b0905c4737ac82383705d710d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:49.456503 7949 tablet_service.cc:1467] Processing CreateTablet for tablet ca5075b0905c4737ac82383705d710d5 (DEFAULT_TABLE table=test-table [id=fd381de0209a45bd9693945778bb22ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:49.457464 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5075b0905c4737ac82383705d710d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:49.464833 8032 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Bootstrap starting.
I20260928 23:58:49.465775 8034 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Bootstrap starting.
I20260928 23:58:49.466301 8033 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Bootstrap starting.
I20260928 23:58:49.468030 8033 tablet_bootstrap.cc:654] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:49.468047 8032 tablet_bootstrap.cc:654] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:49.468046 8034 tablet_bootstrap.cc:654] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:49.469195 8033 log.cc:824] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:49.469199 8032 log.cc:824] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:49.469332 8034 log.cc:824] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:49.471495 8033 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: No bootstrap required, opened a new log
I20260928 23:58:49.471729 8033 ts_tablet_manager.cc:1397] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:49.471961 8034 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: No bootstrap required, opened a new log
I20260928 23:58:49.471961 8032 tablet_bootstrap.cc:492] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: No bootstrap required, opened a new log
I20260928 23:58:49.472164 8034 ts_tablet_manager.cc:1397] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:58:49.472164 8032 ts_tablet_manager.cc:1397] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 23:58:49.475188 8033 raft_consensus.cc:348] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.475512 8033 raft_consensus.cc:374] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:49.475570 8033 raft_consensus.cc:729] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f068b159a2478eb0f1197c04217552, State: Initialized, Role: FOLLOWER
I20260928 23:58:49.476051 8033 consensus_queue.cc:260] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.476545 8034 raft_consensus.cc:348] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.476876 8034 raft_consensus.cc:374] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:49.476933 8034 raft_consensus.cc:729] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9a07d5791442fdbe4855f3d189fc50, State: Initialized, Role: FOLLOWER
I20260928 23:58:49.477218 8033 ts_tablet_manager.cc:1428] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:58:49.477102 8032 raft_consensus.cc:348] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.477389 8032 raft_consensus.cc:374] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:49.477442 8032 raft_consensus.cc:729] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d464a929631143c0a2d7aaff4a5bd44a, State: Initialized, Role: FOLLOWER
I20260928 23:58:49.477439 8034 consensus_queue.cc:260] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.477600 7881 heartbeater.cc:503] Master 127.20.88.62:44491 was elected leader, sending a full tablet report...
I20260928 23:58:49.477927 8032 consensus_queue.cc:260] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.478643 7747 heartbeater.cc:503] Master 127.20.88.62:44491 was elected leader, sending a full tablet report...
I20260928 23:58:49.478899 8032 ts_tablet_manager.cc:1428] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:58:49.478503 8034 ts_tablet_manager.cc:1428] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:58:49.479902 8015 heartbeater.cc:503] Master 127.20.88.62:44491 was elected leader, sending a full tablet report...
I20260928 23:58:49.616890 8038 raft_consensus.cc:482] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:49.617169 8038 raft_consensus.cc:504] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.618453 8038 leader_election.cc:288] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9a07d5791442fdbe4855f3d189fc50 (127.20.88.3:40513), d464a929631143c0a2d7aaff4a5bd44a (127.20.88.1:39963)
I20260928 23:58:49.623790 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.619057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:35265 (local address 127.20.88.3:40513)
0928 23:58:49.619207 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.619210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.619729 (+ 519us) server_negotiation.cc:407] Connection header received
0928 23:58:49.619790 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.619793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.619848 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.619937 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:49.620195 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.620857 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.622372 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.623000 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.623118 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.623188 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.623558 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.623648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:58:49.623790 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.618974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38687 (local address 127.20.88.1:39963)
0928 23:58:49.619138 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.619145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.619167 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:58:49.619227 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.619231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.619304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.619390 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:49.619753 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.620582 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.622299 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.623000 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.623118 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.623188 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.623558 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.623648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:58:49.623822 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.618781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39963 (local address 127.20.88.2:38687)
0928 23:58:49.618946 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.618964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.619100 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.619438 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.619441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.619464 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:49.619591 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.619601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.620721 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.620723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.622129 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.622141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.623165 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.623169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.623418 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.623435 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.623505 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.623632 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":87}
I20260928 23:58:49.624336 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.618786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:40513 (local address 127.20.88.2:35265)
0928 23:58:49.619568 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.619582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:49.619701 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:49.619961 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:49.619963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:49.619978 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:49.620078 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.620082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.621068 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.621071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.622243 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:49.622251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.623165 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.623169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:49.623339 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.623419 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 23:58:49.623505 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:58:49.623632 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":490,"thread_start_us":114,"threads_started":1}
I20260928 23:58:49.624709 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5075b0905c4737ac82383705d710d5" candidate_uuid: "30f068b159a2478eb0f1197c04217552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d464a929631143c0a2d7aaff4a5bd44a" is_pre_election: true
I20260928 23:58:49.625010 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5075b0905c4737ac82383705d710d5" candidate_uuid: "30f068b159a2478eb0f1197c04217552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a07d5791442fdbe4855f3d189fc50" is_pre_election: true
I20260928 23:58:49.625051 7701 raft_consensus.cc:2454] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f068b159a2478eb0f1197c04217552 in term 0.
I20260928 23:58:49.625311 7969 raft_consensus.cc:2454] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f068b159a2478eb0f1197c04217552 in term 0.
I20260928 23:58:49.625587 7768 leader_election.cc:302] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f068b159a2478eb0f1197c04217552, d464a929631143c0a2d7aaff4a5bd44a; no voters:
I20260928 23:58:49.625937 8038 raft_consensus.cc:2781] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:49.626044 8038 raft_consensus.cc:482] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:49.626102 8038 raft_consensus.cc:3037] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:49.627228 8038 raft_consensus.cc:504] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.627635 8038 leader_election.cc:288] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [CANDIDATE]: Term 1 election: Requested vote from peers 5a9a07d5791442fdbe4855f3d189fc50 (127.20.88.3:40513), d464a929631143c0a2d7aaff4a5bd44a (127.20.88.1:39963)
I20260928 23:58:49.628082 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5075b0905c4737ac82383705d710d5" candidate_uuid: "30f068b159a2478eb0f1197c04217552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d464a929631143c0a2d7aaff4a5bd44a"
I20260928 23:58:49.628095 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5075b0905c4737ac82383705d710d5" candidate_uuid: "30f068b159a2478eb0f1197c04217552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9a07d5791442fdbe4855f3d189fc50"
I20260928 23:58:49.628252 7969 raft_consensus.cc:3037] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:49.628252 7701 raft_consensus.cc:3037] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:49.629671 7701 raft_consensus.cc:2454] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f068b159a2478eb0f1197c04217552 in term 1.
I20260928 23:58:49.629671 7969 raft_consensus.cc:2454] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f068b159a2478eb0f1197c04217552 in term 1.
I20260928 23:58:49.630091 7770 leader_election.cc:302] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30f068b159a2478eb0f1197c04217552, 5a9a07d5791442fdbe4855f3d189fc50, d464a929631143c0a2d7aaff4a5bd44a; no voters:
I20260928 23:58:49.630285 8038 raft_consensus.cc:2781] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:49.630513 8038 raft_consensus.cc:686] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 1 LEADER]: Becoming Leader. State: Replica: 30f068b159a2478eb0f1197c04217552, State: Running, Role: LEADER
I20260928 23:58:49.630899 8038 consensus_queue.cc:237] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } }
I20260928 23:58:49.634532 7554 catalog_manager.cc:5495] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30f068b159a2478eb0f1197c04217552 (127.20.88.2). New cstate: current_term: 1 leader_uuid: "30f068b159a2478eb0f1197c04217552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f068b159a2478eb0f1197c04217552" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 35239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 } health_report { overall_health: UNKNOWN } } }
W20260928 23:58:49.645656 8016 tablet.cc:2369] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:58:49.651485 7882 tablet.cc:2369] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:58:49.658475 7748 tablet.cc:2369] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:49.691533 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.688009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.20.88.2:35239)
0928 23:58:49.688143 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.688147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.688162 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:58:49.688335 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.688338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.688399 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.688498 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:49.688755 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.689251 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.690133 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.690368 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.690449 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.690715 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:49.691166 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:49.691261 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.691341 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.691396 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:58:49.697825 7969 raft_consensus.cc:1261] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 1 FOLLOWER]: Refusing update from remote peer 30f068b159a2478eb0f1197c04217552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:49.697825 7701 raft_consensus.cc:1261] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 1 FOLLOWER]: Refusing update from remote peer 30f068b159a2478eb0f1197c04217552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:49.698638 8038 consensus_queue.cc:1035] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9a07d5791442fdbe4855f3d189fc50" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:49.698839 8042 consensus_queue.cc:1035] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d464a929631143c0a2d7aaff4a5bd44a" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:49.715943 7555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:49.716146 7555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:49.718731 7555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59abeb6ae5394409a3e1d8d9350ddd55: Sending DeleteTablet for 3 replicas of tablet ca5075b0905c4737ac82383705d710d5
I20260928 23:58:49.719604 7949 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5075b0905c4737ac82383705d710d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:49 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260928 23:58:49.719600 7815 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5075b0905c4737ac82383705d710d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:49 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 23:58:49.719965 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5075b0905c4737ac82383705d710d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:49 UTC) from {username='kudu', principal='kudu/127.20.88.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260928 23:58:49.720170 8054 tablet_replica.cc:330] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: stopping tablet replica
I20260928 23:58:49.720551 8054 raft_consensus.cc:2229] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:49.720984 8056 tablet_replica.cc:330] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a: stopping tablet replica
I20260928 23:58:49.721025 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7618
I20260928 23:58:49.721387 8056 raft_consensus.cc:2229] T ca5075b0905c4737ac82383705d710d5 P d464a929631143c0a2d7aaff4a5bd44a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:49.721666 8055 tablet_replica.cc:330] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: stopping tablet replica
I20260928 23:58:49.721076 8054 raft_consensus.cc:2258] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:49.722052 8055 raft_consensus.cc:2229] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:49.722421 8055 raft_consensus.cc:2258] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:49.723712 8054 ts_tablet_manager.cc:1905] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:49.724068 8055 ts_tablet_manager.cc:1905] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:49.727295 8055 ts_tablet_manager.cc:1918] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:49.727516 8055 log.cc:1196] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/ca5075b0905c4737ac82383705d710d5
I20260928 23:58:49.727494 8054 ts_tablet_manager.cc:1918] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:49.727600 8054 log.cc:1196] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/ca5075b0905c4737ac82383705d710d5
I20260928 23:58:49.727998 8055 ts_tablet_manager.cc:1939] T ca5075b0905c4737ac82383705d710d5 P 5a9a07d5791442fdbe4855f3d189fc50: Deleting consensus metadata
I20260928 23:58:49.727994 8054 ts_tablet_manager.cc:1939] T ca5075b0905c4737ac82383705d710d5 P 30f068b159a2478eb0f1197c04217552: Deleting consensus metadata
I20260928 23:58:49.729478 7543 catalog_manager.cc:4841] TS 30f068b159a2478eb0f1197c04217552 (127.20.88.2:35239): tablet ca5075b0905c4737ac82383705d710d5 (table test-table [id=fd381de0209a45bd9693945778bb22ae]) successfully deleted
W20260928 23:58:49.729775 7540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:39963 (error 108)
I20260928 23:58:49.729481 7542 catalog_manager.cc:4841] TS 5a9a07d5791442fdbe4855f3d189fc50 (127.20.88.3:40513): tablet ca5075b0905c4737ac82383705d710d5 (table test-table [id=fd381de0209a45bd9693945778bb22ae]) successfully deleted
I20260928 23:58:49.730654 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7752
I20260928 23:58:49.731011 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.730670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:39963 (local address 127.0.0.1:53696)
0928 23:58:49.730831 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:49.730908 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 23:58:49.731724 7540 catalog_manager.cc:4584] TS d464a929631143c0a2d7aaff4a5bd44a (127.20.88.1:39963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca5075b0905c4737ac82383705d710d5: Network error: Client connection negotiation failed: client connection to 127.20.88.1:39963: connect: Connection refused (error 111)
I20260928 23:58:49.736990 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7886
I20260928 23:58:49.743302 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 7524
2026-09-28T23:58:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2529 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T23:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:49Z Disabled control of system clock
I20260928 23:58:49.775969 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37733
--webserver_interface=127.20.88.62
--webserver_port=0
--builtin_ntp_servers=127.20.88.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:49.894961 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:49.895300 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:49.895344 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:49.899781 8061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:49.899876 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:49.899893 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:49.899911 8061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:49.899927 8061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:49.905330 8061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:49.906509 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:49.908003 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:49.917758 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:49.917760 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:49.917758 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:49.918172 8061 server_base.cc:1034] running on GCE node
I20260928 23:58:49.918808 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:49.920018 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:49.921214 8061 hybrid_clock.cc:648] HybridClock initialized: now 1790639929921186 us; error 59 us; skew 500 ppm
I20260928 23:58:49.923334 8061 webserver.cc:460] Webserver started at http://127.20.88.62:45629/ using document root <none> and password file <none>
I20260928 23:58:49.923985 8061 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:49.924084 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:49.924389 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:49.926640 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "cb36822b09174acd921ddff1a52518d7"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "84fb0e72bbfc48c6d873c58212526088"
server_key_iv: "c15febeeaa89809e04503910cd597cbe"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.927263 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "cb36822b09174acd921ddff1a52518d7"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "84fb0e72bbfc48c6d873c58212526088"
server_key_iv: "c15febeeaa89809e04503910cd597cbe"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.931318 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 23:58:49.934114 8075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:49.935496 8061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:49.935662 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "cb36822b09174acd921ddff1a52518d7"
format_stamp: "Formatted at 2026-09-28 23:58:49 on dist-test-slave-wvpw"
server_key: "84fb0e72bbfc48c6d873c58212526088"
server_key_iv: "c15febeeaa89809e04503910cd597cbe"
server_key_version: "encryptionkey@0"
I20260928 23:58:49.935793 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:49.950212 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:49.951009 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:49.951231 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:49.958894 8061 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:37733
I20260928 23:58:49.958904 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:37733 every 8 connection(s)
I20260928 23:58:49.959899 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:49.963088 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8061
I20260928 23:58:49.963254 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:49.963357 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:49.963637 20832 external_mini_cluster.cc:1428] Setting key aed1245891d662ecf259efa838784aa2
I20260928 23:58:49.970551 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: Bootstrap starting.
I20260928 23:58:49.973205 8128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:49.973984 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:49.965178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.20.88.62:37733)
0928 23:58:49.965970 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:49.965987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:49.966024 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:58:49.966925 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:49.966946 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:49.967390 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:49.967764 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:49.968321 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.969332 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.970293 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:49.970878 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:49.971068 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:49.972021 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:49.972026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:49.972040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:49.972519 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:49.972581 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:49.972602 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:49.972715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:49.972836 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:58:49.973136 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":161,"threads_started":1}
I20260928 23:58:49.974009 8128 log.cc:824] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:49.976336 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: No bootstrap required, opened a new log
I20260928 23:58:49.980006 8128 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } }
I20260928 23:58:49.980361 8128 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:49.980453 8128 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb36822b09174acd921ddff1a52518d7, State: Initialized, Role: FOLLOWER
I20260928 23:58:49.980962 8128 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } }
I20260928 23:58:49.981125 8128 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:49.981196 8128 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:49.981333 8128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:49.982605 8128 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } }
I20260928 23:58:49.983008 8128 leader_election.cc:302] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb36822b09174acd921ddff1a52518d7; no voters:
I20260928 23:58:49.983359 8128 leader_election.cc:288] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:49.983520 8133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:49.983775 8133 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [term 1 LEADER]: Becoming Leader. State: Replica: cb36822b09174acd921ddff1a52518d7, State: Running, Role: LEADER
I20260928 23:58:49.984109 8133 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } }
I20260928 23:58:49.984704 8128 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:49.985914 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb36822b09174acd921ddff1a52518d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } } }
I20260928 23:58:49.986047 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:49.986349 8135 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb36822b09174acd921ddff1a52518d7. Latest consensus state: current_term: 1 leader_uuid: "cb36822b09174acd921ddff1a52518d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb36822b09174acd921ddff1a52518d7" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37733 } } }
I20260928 23:58:49.987423 8135 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7 [sys.catalog]: This master's current role is: LEADER
W20260928 23:58:49.989893 8148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:58:49.989975 8148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:58:49.990068 8142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:49.990840 8142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:49.997040 8142 catalog_manager.cc:1347] Generated new cluster ID: 38fb6ee8d8714e3aac724088a8fd3aff
I20260928 23:58:49.997149 8142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:50.035698 8142 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:50.036888 8142 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:50.043644 8142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: Generated new TSK 0
I20260928 23:58:50.044620 8142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:50.051684 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.1:0
--local_ip_for_outbound_sockets=127.20.88.1
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--builtin_ntp_servers=127.20.88.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:58:50.179260 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:50.179564 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:50.179606 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:50.183735 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:50.183810 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:50.183869 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.1
I20260928 23:58:50.189662 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.88.1
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.1
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:50.190809 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:50.192194 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:50.203106 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:50.203285 8152 server_base.cc:1034] running on GCE node
W20260928 23:58:50.203096 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:50.203101 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:50.203904 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:50.204684 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:50.205873 8152 hybrid_clock.cc:648] HybridClock initialized: now 1790639930205856 us; error 30 us; skew 500 ppm
I20260928 23:58:50.215248 8152 webserver.cc:460] Webserver started at http://127.20.88.1:41599/ using document root <none> and password file <none>
I20260928 23:58:50.216003 8152 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:50.216080 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:50.216398 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:50.218668 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data/instance:
uuid: "ab35edf5111a4b9aa6324a72460147b8"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "51098a5d68d12a5d487e0cada99c85be"
server_key_iv: "c878ab81bb605a03ce03a28c40607b26"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.219276 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance:
uuid: "ab35edf5111a4b9aa6324a72460147b8"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "51098a5d68d12a5d487e0cada99c85be"
server_key_iv: "c878ab81bb605a03ce03a28c40607b26"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.223419 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:58:50.226264 8166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.227780 8152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:58:50.227962 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal
uuid: "ab35edf5111a4b9aa6324a72460147b8"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "51098a5d68d12a5d487e0cada99c85be"
server_key_iv: "c878ab81bb605a03ce03a28c40607b26"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.228104 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:50.259487 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:50.260408 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:50.260648 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:50.261307 8152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:50.262367 8152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:50.262451 8152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.262529 8152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:50.262580 8152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.273211 8152 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.1:42199
I20260928 23:58:50.273257 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.1:42199 every 8 connection(s)
I20260928 23:58:50.274281 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/data/info.pb
I20260928 23:58:50.281230 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8152
I20260928 23:58:50.281352 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-0/wal/instance
I20260928 23:58:50.281670 20832 external_mini_cluster.cc:1428] Setting key 7b23a07742fb00776254268783b6af94
I20260928 23:58:50.283841 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.2:0
--local_ip_for_outbound_sockets=127.20.88.2
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--builtin_ntp_servers=127.20.88.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:50.285292 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.276430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.1:39709 (local address 127.20.88.62:37733)
0928 23:58:50.276582 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.276586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.277744 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 23:58:50.279315 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.279319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.279385 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.279503 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:50.280575 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.281139 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.282676 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.282925 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.283075 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.284689 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:50.284693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:50.284696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:50.284719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:50.284748 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:50.284757 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:50.284849 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.285037 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.285118 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:58:50.286541 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.276862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37733 (local address 127.20.88.1:39709)
0928 23:58:50.277573 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.277619 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.279055 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.279704 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.279712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.279816 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:50.280418 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.280435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.281291 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.281297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.282545 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.282558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.283162 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.283166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.283392 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.284202 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:50.284204 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:50.284408 (+ 204us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:50.284504 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:50.284852 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:50.284864 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:50.284871 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.285021 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.285528 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260928 23:58:50.288029 8280 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37733
I20260928 23:58:50.288407 8280 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:50.289224 8280 heartbeater.cc:511] Master 127.20.88.62:37733 requested a full tablet report, sending...
I20260928 23:58:50.291181 8092 ts_manager.cc:194] Registered new tserver with Master: ab35edf5111a4b9aa6324a72460147b8 (127.20.88.1:42199)
I20260928 23:58:50.292635 8092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.1:39709
W20260928 23:58:50.405206 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:50.405525 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:50.405567 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:50.409901 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:50.409986 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:50.410048 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.2
I20260928 23:58:50.415417 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.88.2
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.2
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:50.416803 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:50.418350 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:50.428896 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:50.428922 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:50.428918 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:50.430095 8283 server_base.cc:1034] running on GCE node
I20260928 23:58:50.430578 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:50.431485 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:50.432651 8283 hybrid_clock.cc:648] HybridClock initialized: now 1790639930432610 us; error 56 us; skew 500 ppm
I20260928 23:58:50.434703 8283 webserver.cc:460] Webserver started at http://127.20.88.2:32907/ using document root <none> and password file <none>
I20260928 23:58:50.435333 8283 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:50.435400 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:50.435669 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:50.438050 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data/instance:
uuid: "4bd8e8d8960f4ac69948e976d3e88fcf"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "2316b4d9652c04b838db57334ed69405"
server_key_iv: "664b92429d4fa66c7aa85ecbe38a6f6a"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.438656 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance:
uuid: "4bd8e8d8960f4ac69948e976d3e88fcf"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "2316b4d9652c04b838db57334ed69405"
server_key_iv: "664b92429d4fa66c7aa85ecbe38a6f6a"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.442993 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:50.445915 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.447431 8283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:50.447602 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal
uuid: "4bd8e8d8960f4ac69948e976d3e88fcf"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "2316b4d9652c04b838db57334ed69405"
server_key_iv: "664b92429d4fa66c7aa85ecbe38a6f6a"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.447757 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:50.465283 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:50.466076 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:50.466320 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:50.466995 8283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:50.468065 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:50.468140 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.468217 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:50.468338 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 23:58:50.478282 8283 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.2:45659
I20260928 23:58:50.478304 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.2:45659 every 8 connection(s)
I20260928 23:58:50.479312 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/data/info.pb
I20260928 23:58:50.481968 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8283
I20260928 23:58:50.482101 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal/instance
I20260928 23:58:50.482450 20832 external_mini_cluster.cc:1428] Setting key 093c9ef34f062e9212f17d1964fcbe2f
I20260928 23:58:50.484984 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.88.3:0
--local_ip_for_outbound_sockets=127.20.88.3
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--builtin_ntp_servers=127.20.88.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:58:50.490633 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.481409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:41605 (local address 127.20.88.62:37733)
0928 23:58:50.481576 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.481580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.483015 (+ 1435us) server_negotiation.cc:407] Connection header received
0928 23:58:50.484957 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.484961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.485024 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.485122 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:50.486373 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.487043 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.488438 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.488695 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.488830 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.490078 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:50.490083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:50.490085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:50.490101 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:50.490131 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:50.490140 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:50.490219 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.490398 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.490476 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 23:58:50.491672 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.481966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37733 (local address 127.20.88.2:41605)
0928 23:58:50.482799 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.482871 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.484663 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.485346 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.485360 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.485483 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:50.486219 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.486235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.487170 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.487174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.488226 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.488291 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.488875 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.488879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.489077 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.489783 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:50.489784 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:50.489832 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:50.489908 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:50.490232 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:50.490243 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:50.490249 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.490398 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.490722 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":170,"threads_started":1}
I20260928 23:58:50.493269 8411 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37733
I20260928 23:58:50.493553 8411 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:50.494211 8411 heartbeater.cc:511] Master 127.20.88.62:37733 requested a full tablet report, sending...
I20260928 23:58:50.495507 8092 ts_manager.cc:194] Registered new tserver with Master: 4bd8e8d8960f4ac69948e976d3e88fcf (127.20.88.2:45659)
I20260928 23:58:50.496388 8092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.2:41605
W20260928 23:58:50.605670 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:50.605971 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:50.606011 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:50.610181 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:50.610251 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:50.610312 8414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.88.3
I20260928 23:58:50.615351 8414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.88.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.88.3
--webserver_port=0
--tserver_master_addrs=127.20.88.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.88.3
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:50.616644 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:50.618043 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:58:50.628168 8414 server_base.cc:1034] running on GCE node
W20260928 23:58:50.628134 8419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:50.628228 8422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:50.628245 8420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:50.628916 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:50.629633 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:50.630857 8414 hybrid_clock.cc:648] HybridClock initialized: now 1790639930630837 us; error 35 us; skew 500 ppm
I20260928 23:58:50.633126 8414 webserver.cc:460] Webserver started at http://127.20.88.3:36899/ using document root <none> and password file <none>
I20260928 23:58:50.633764 8414 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:50.633883 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:50.634133 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:50.636427 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data/instance:
uuid: "8687453834714aab967ca3d82b6efc28"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "17a8ff814d50f731be6a87dce19b9331"
server_key_iv: "12966833e4f4005d809303f74a8ce480"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.637009 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance:
uuid: "8687453834714aab967ca3d82b6efc28"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "17a8ff814d50f731be6a87dce19b9331"
server_key_iv: "12966833e4f4005d809303f74a8ce480"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.641438 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:58:50.644390 8428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.645737 8414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:50.645933 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal
uuid: "8687453834714aab967ca3d82b6efc28"
format_stamp: "Formatted at 2026-09-28 23:58:50 on dist-test-slave-wvpw"
server_key: "17a8ff814d50f731be6a87dce19b9331"
server_key_iv: "12966833e4f4005d809303f74a8ce480"
server_key_version: "encryptionkey@0"
I20260928 23:58:50.646067 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:50.660401 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:50.661195 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:50.661406 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:50.662037 8414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:58:50.663044 8414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:58:50.663117 8414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.663192 8414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:58:50.663242 8414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:50.672729 8414 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.3:36617
I20260928 23:58:50.672813 8541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.3:36617 every 8 connection(s)
I20260928 23:58:50.673735 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/data/info.pb
I20260928 23:58:50.682498 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8414
I20260928 23:58:50.682723 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal/instance
I20260928 23:58:50.683144 20832 external_mini_cluster.cc:1428] Setting key 3d82d5ab677add1b9440adf6cbb1b91b
I20260928 23:58:50.683671 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.675885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.3:47155 (local address 127.20.88.62:37733)
0928 23:58:50.676037 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.676041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.677119 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 23:58:50.678490 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.678493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.678548 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.678648 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:50.679637 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.680217 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.681459 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.681683 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.681776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.683046 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:50.683052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:50.683057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:50.683074 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:50.683105 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:50.683118 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:50.683255 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.683457 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.683533 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:58:50.684842 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.676261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37733 (local address 127.20.88.3:47155)
0928 23:58:50.676943 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.676984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.678264 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.678858 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.678866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.678950 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:58:50.679477 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.679492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.680429 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.680431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.681318 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.681328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.681834 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.681837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.681982 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.682680 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:58:50.682682 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:58:50.682740 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:58:50.682858 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:58:50.683278 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:58:50.683289 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:58:50.683295 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.683417 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.683725 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260928 23:58:50.686110 8542 heartbeater.cc:347] Connected to a master server at 127.20.88.62:37733
I20260928 23:58:50.686412 8542 heartbeater.cc:464] Registering TS with master...
I20260928 23:58:50.686995 8542 heartbeater.cc:511] Master 127.20.88.62:37733 requested a full tablet report, sending...
I20260928 23:58:50.688122 8091 ts_manager.cc:194] Registered new tserver with Master: 8687453834714aab967ca3d82b6efc28 (127.20.88.3:36617)
I20260928 23:58:50.688827 8091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.88.3:47155
I20260928 23:58:50.698169 20832 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:58:50.704988 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.700705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.20.88.62:37733)
0928 23:58:50.700939 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.700945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.701122 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:58:50.701329 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.701332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.701391 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.701484 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:50.701813 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.702351 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.703254 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.703506 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.703630 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.704536 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:50.704539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:50.704541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:50.704554 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:50.704580 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:50.704588 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:50.704651 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.704782 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.704863 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 23:58:50.708602 8091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:58:50.712163 8091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:58:50.728173 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.722515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.20.88.1:42199)
0928 23:58:50.722938 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.722952 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.722980 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:58:50.723109 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.723116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.723271 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.723459 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:50.723594 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.724720 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.726586 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.727310 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.727415 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.727612 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.727920 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.728032 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
I20260928 23:58:50.728163 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.722605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:36617 (local address 127.0.0.1:47280)
0928 23:58:50.723168 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.723179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.723247 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.724101 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.724105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.724121 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:50.724208 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.724218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.725511 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.725517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.726379 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.726389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.727441 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.727445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.727637 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.727733 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.727849 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.727993 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":105,"thread_start_us":88,"threads_started":1}
I20260928 23:58:50.728590 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.722795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.20.88.3:36617)
0928 23:58:50.723735 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.723744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.723761 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:58:50.723817 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.723821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.723956 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.724106 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:50.724357 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.725401 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.726504 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.727296 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.727416 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.727612 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.728325 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.728432 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":187,"threads_started":1}
I20260928 23:58:50.728633 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.722261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42199 (local address 127.0.0.1:38034)
0928 23:58:50.722895 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.722909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.722987 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.723441 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.723443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.723458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:50.723529 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.723532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.724842 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.724844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.726476 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.726488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.727441 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.727445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.727640 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.727734 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.728427 (+ 693us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.728506 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":182,"thread_start_us":123,"threads_started":1}
I20260928 23:58:50.729341 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.721556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:45659 (local address 127.0.0.1:40388)
0928 23:58:50.722040 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.722097 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.722229 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.723054 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.723060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.723085 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:50.723229 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.723237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.725044 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.725047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.726479 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.726488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.728935 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.728939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.729042 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.729069 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.729144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.729209 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":68,"thread_start_us":80,"threads_started":1}
I20260928 23:58:50.730115 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.721687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.20.88.2:45659)
0928 23:58:50.722183 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.722197 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.722228 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:58:50.722450 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.722456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.722680 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.722870 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:50.723356 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.724661 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.727685 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.728552 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.728649 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.728864 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.729903 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.729988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":144,"threads_started":1}
I20260928 23:58:50.732272 8476 tablet_service.cc:1467] Processing CreateTablet for tablet 5a201ba87b704ae48a620b26caca1a5a (DEFAULT_TABLE table=test-table [id=06f22f0a50964040b5485bcbdd8da697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:50.732990 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 5a201ba87b704ae48a620b26caca1a5a (DEFAULT_TABLE table=test-table [id=06f22f0a50964040b5485bcbdd8da697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:50.733578 8476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a201ba87b704ae48a620b26caca1a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:50.734018 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a201ba87b704ae48a620b26caca1a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:50.738669 8214 tablet_service.cc:1467] Processing CreateTablet for tablet 5a201ba87b704ae48a620b26caca1a5a (DEFAULT_TABLE table=test-table [id=06f22f0a50964040b5485bcbdd8da697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:58:50.739934 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a201ba87b704ae48a620b26caca1a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:50.742211 8559 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Bootstrap starting.
I20260928 23:58:50.743986 8560 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Bootstrap starting.
I20260928 23:58:50.745471 8559 tablet_bootstrap.cc:654] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:50.746284 8560 tablet_bootstrap.cc:654] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:50.746408 8559 log.cc:824] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:50.747145 8560 log.cc:824] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:50.747270 8562 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Bootstrap starting.
I20260928 23:58:50.748857 8559 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: No bootstrap required, opened a new log
I20260928 23:58:50.749073 8559 ts_tablet_manager.cc:1397] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:58:50.749203 8560 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: No bootstrap required, opened a new log
I20260928 23:58:50.749392 8560 ts_tablet_manager.cc:1397] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:58:50.750459 8562 tablet_bootstrap.cc:654] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:50.751362 8562 log.cc:824] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:50.753439 8560 raft_consensus.cc:348] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.753697 8560 raft_consensus.cc:374] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:50.753707 8562 tablet_bootstrap.cc:492] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: No bootstrap required, opened a new log
I20260928 23:58:50.753753 8560 raft_consensus.cc:729] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8687453834714aab967ca3d82b6efc28, State: Initialized, Role: FOLLOWER
I20260928 23:58:50.753638 8559 raft_consensus.cc:348] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.753906 8562 ts_tablet_manager.cc:1397] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:58:50.753921 8559 raft_consensus.cc:374] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:50.753969 8559 raft_consensus.cc:729] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd8e8d8960f4ac69948e976d3e88fcf, State: Initialized, Role: FOLLOWER
I20260928 23:58:50.754266 8560 consensus_queue.cc:260] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.754462 8559 consensus_queue.cc:260] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.755050 8542 heartbeater.cc:503] Master 127.20.88.62:37733 was elected leader, sending a full tablet report...
I20260928 23:58:50.755383 8560 ts_tablet_manager.cc:1428] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:58:50.755435 8559 ts_tablet_manager.cc:1428] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:58:50.755775 8411 heartbeater.cc:503] Master 127.20.88.62:37733 was elected leader, sending a full tablet report...
I20260928 23:58:50.758230 8562 raft_consensus.cc:348] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.758502 8562 raft_consensus.cc:374] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:50.758566 8562 raft_consensus.cc:729] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab35edf5111a4b9aa6324a72460147b8, State: Initialized, Role: FOLLOWER
I20260928 23:58:50.759115 8562 consensus_queue.cc:260] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.759831 8280 heartbeater.cc:503] Master 127.20.88.62:37733 was elected leader, sending a full tablet report...
I20260928 23:58:50.760260 8562 ts_tablet_manager.cc:1428] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260928 23:58:50.775648 8281 tablet.cc:2369] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:58:50.850746 8566 raft_consensus.cc:482] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:58:50.850956 8566 raft_consensus.cc:504] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.852146 8566 leader_election.cc:288] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab35edf5111a4b9aa6324a72460147b8 (127.20.88.1:42199), 8687453834714aab967ca3d82b6efc28 (127.20.88.3:36617)
I20260928 23:58:50.856395 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.852521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:38221 (local address 127.20.88.1:42199)
0928 23:58:50.852657 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.852661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.852677 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:58:50.852760 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.852763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.852832 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.852907 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:50.853320 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.853955 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.855139 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.855788 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.855890 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.855943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.856154 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.856288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:58:50.856383 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.852380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42199 (local address 127.20.88.2:38221)
0928 23:58:50.852532 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.852545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.852661 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.853020 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.853023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.853044 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:50.853199 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.853207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.854067 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.854070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.854994 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.855003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.855915 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.855919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.856022 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.856048 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.856127 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.856205 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 23:58:50.857146 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.852717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.3:36617 (local address 127.20.88.2:35941)
0928 23:58:50.853058 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.853069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:58:50.853171 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:58:50.853421 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:58:50.853423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:58:50.853440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:58:50.853549 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.853555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.854379 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.854382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.855613 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:58:50.855624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.856742 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.856746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:58:50.856838 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.856864 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:58:50.856942 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:58:50.857007 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260928 23:58:50.857137 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.852851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.88.2:35941 (local address 127.20.88.3:36617)
0928 23:58:50.852979 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.852986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.853141 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:58:50.853264 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.853268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.853325 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.853392 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:58:50.853650 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.854257 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.855829 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.856465 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.856572 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.856634 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.856960 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.857034 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:58:50.857173 8234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a201ba87b704ae48a620b26caca1a5a" candidate_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab35edf5111a4b9aa6324a72460147b8" is_pre_election: true
I20260928 23:58:50.857569 8234 raft_consensus.cc:2454] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd8e8d8960f4ac69948e976d3e88fcf in term 0.
I20260928 23:58:50.857740 8496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a201ba87b704ae48a620b26caca1a5a" candidate_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8687453834714aab967ca3d82b6efc28" is_pre_election: true
I20260928 23:58:50.858047 8496 raft_consensus.cc:2454] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd8e8d8960f4ac69948e976d3e88fcf in term 0.
I20260928 23:58:50.858161 8298 leader_election.cc:302] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd8e8d8960f4ac69948e976d3e88fcf, ab35edf5111a4b9aa6324a72460147b8; no voters:
I20260928 23:58:50.858529 8566 raft_consensus.cc:2781] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:58:50.858631 8566 raft_consensus.cc:482] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:58:50.858688 8566 raft_consensus.cc:3037] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:50.859721 8566 raft_consensus.cc:504] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.860088 8566 leader_election.cc:288] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [CANDIDATE]: Term 1 election: Requested vote from peers ab35edf5111a4b9aa6324a72460147b8 (127.20.88.1:42199), 8687453834714aab967ca3d82b6efc28 (127.20.88.3:36617)
I20260928 23:58:50.860534 8234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a201ba87b704ae48a620b26caca1a5a" candidate_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab35edf5111a4b9aa6324a72460147b8"
I20260928 23:58:50.860599 8496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a201ba87b704ae48a620b26caca1a5a" candidate_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8687453834714aab967ca3d82b6efc28"
I20260928 23:58:50.860674 8234 raft_consensus.cc:3037] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:50.860751 8496 raft_consensus.cc:3037] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:50.861810 8496 raft_consensus.cc:2454] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd8e8d8960f4ac69948e976d3e88fcf in term 1.
I20260928 23:58:50.861820 8234 raft_consensus.cc:2454] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd8e8d8960f4ac69948e976d3e88fcf in term 1.
I20260928 23:58:50.862144 8298 leader_election.cc:302] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd8e8d8960f4ac69948e976d3e88fcf, 8687453834714aab967ca3d82b6efc28; no voters:
I20260928 23:58:50.862368 8566 raft_consensus.cc:2781] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:50.862619 8566 raft_consensus.cc:686] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 1 LEADER]: Becoming Leader. State: Replica: 4bd8e8d8960f4ac69948e976d3e88fcf, State: Running, Role: LEADER
I20260928 23:58:50.862969 8566 consensus_queue.cc:237] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } }
I20260928 23:58:50.866237 8091 catalog_manager.cc:5495] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd8e8d8960f4ac69948e976d3e88fcf (127.20.88.2). New cstate: current_term: 1 leader_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bd8e8d8960f4ac69948e976d3e88fcf" member_type: VOTER last_known_addr { host: "127.20.88.2" port: 45659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 } health_report { overall_health: UNKNOWN } } }
I20260928 23:58:50.888672 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.884760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.20.88.2:45659)
0928 23:58:50.884981 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:50.884986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:50.884999 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:58:50.885088 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:50.885090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:50.885144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:50.885289 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:58:50.885540 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.886160 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.887066 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:50.887298 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:50.887408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:50.887744 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:58:50.888185 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:58:50.888385 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:50.888456 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:58:50.888514 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:58:50.895140 8496 raft_consensus.cc:1261] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 1 FOLLOWER]: Refusing update from remote peer 4bd8e8d8960f4ac69948e976d3e88fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:50.895140 8234 raft_consensus.cc:1261] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 1 FOLLOWER]: Refusing update from remote peer 4bd8e8d8960f4ac69948e976d3e88fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:58:50.895990 8566 consensus_queue.cc:1035] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8687453834714aab967ca3d82b6efc28" member_type: VOTER last_known_addr { host: "127.20.88.3" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:50.896317 8569 consensus_queue.cc:1035] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab35edf5111a4b9aa6324a72460147b8" member_type: VOTER last_known_addr { host: "127.20.88.1" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:58:50.905462 8574 mvcc.cc:205] Tried to move back new op lower bound from 7334461156938145792 to 7334461156817276928. Current Snapshot: MvccSnapshot[applied={T|T < 7334461156938145792}]
I20260928 23:58:50.915117 8092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:50.915333 8092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:58:50.919386 8092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb36822b09174acd921ddff1a52518d7: Sending DeleteTablet for 3 replicas of tablet 5a201ba87b704ae48a620b26caca1a5a
I20260928 23:58:50.920640 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a201ba87b704ae48a620b26caca1a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:50 UTC) from {username='slave'} at 127.0.0.1:40388
I20260928 23:58:50.920670 8214 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a201ba87b704ae48a620b26caca1a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:50 UTC) from {username='slave'} at 127.0.0.1:38034
I20260928 23:58:50.920878 8476 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a201ba87b704ae48a620b26caca1a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:58:50 UTC) from {username='slave'} at 127.0.0.1:47280
I20260928 23:58:50.921276 8583 tablet_replica.cc:330] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8: stopping tablet replica
I20260928 23:58:50.921535 8584 tablet_replica.cc:330] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: stopping tablet replica
I20260928 23:58:50.921703 8583 raft_consensus.cc:2229] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:50.921959 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8152
I20260928 23:58:50.922000 8584 raft_consensus.cc:2229] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:58:50.922133 8583 raft_consensus.cc:2258] T 5a201ba87b704ae48a620b26caca1a5a P ab35edf5111a4b9aa6324a72460147b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:50.922533 8584 raft_consensus.cc:2258] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:50.922948 8582 tablet_replica.cc:330] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: stopping tablet replica
I20260928 23:58:50.923282 8582 raft_consensus.cc:2229] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:58:50.923746 8582 raft_consensus.cc:2258] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:58:50.924919 8584 ts_tablet_manager.cc:1905] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:50.927064 8582 ts_tablet_manager.cc:1905] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:58:50.928054 8584 ts_tablet_manager.cc:1918] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:50.928270 8584 log.cc:1196] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-2/wal/wals/5a201ba87b704ae48a620b26caca1a5a
I20260928 23:58:50.928681 8584 ts_tablet_manager.cc:1939] T 5a201ba87b704ae48a620b26caca1a5a P 8687453834714aab967ca3d82b6efc28: Deleting consensus metadata
I20260928 23:58:50.930223 8076 catalog_manager.cc:4841] TS 8687453834714aab967ca3d82b6efc28 (127.20.88.3:36617): tablet 5a201ba87b704ae48a620b26caca1a5a (table test-table [id=06f22f0a50964040b5485bcbdd8da697]) successfully deleted
I20260928 23:58:50.930905 8582 ts_tablet_manager.cc:1918] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:58:50.931027 8582 log.cc:1196] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Deleting WAL directory at /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639669102918-20832-0/minicluster-data/ts-1/wal/wals/5a201ba87b704ae48a620b26caca1a5a
I20260928 23:58:50.931391 8582 ts_tablet_manager.cc:1939] T 5a201ba87b704ae48a620b26caca1a5a P 4bd8e8d8960f4ac69948e976d3e88fcf: Deleting consensus metadata
W20260928 23:58:50.932091 8076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.88.1:42199 (error 108)
I20260928 23:58:50.932345 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8283
I20260928 23:58:50.933656 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.933380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.1:42199 (local address 127.0.0.1:38038)
0928 23:58:50.933500 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.933576 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.1:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 23:58:50.934028 8076 catalog_manager.cc:4584] TS ab35edf5111a4b9aa6324a72460147b8 (127.20.88.1:42199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a201ba87b704ae48a620b26caca1a5a: Network error: Client connection negotiation failed: client connection to 127.20.88.1:42199: connect: Connection refused (error 111)
I20260928 23:58:50.938354 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8414
I20260928 23:58:50.940109 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:50.938656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.2:45659 (local address 127.0.0.1:40396)
0928 23:58:50.939902 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0928 23:58:50.939980 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.2:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1105}
I20260928 23:58:50.944419 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8061
2026-09-28T23:58:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1199 ms)
[----------] 6 tests from EncryptionPolicyTest (11094 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:51 dist-test-slave-wvpw krb5kdc[8588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:51 dist-test-slave-wvpw krb5kdc[8588](info): set up 2 sockets
Sep 28 23:58:51 dist-test-slave-wvpw krb5kdc[8588](info): commencing operation
krb5kdc: starting...
W20260928 23:58:51.997113 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:52.128471 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:52.244612 8601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:52.244889 8601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:52.244933 8601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:52.248986 8601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:52.249048 8601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:52.249068 8601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:52.249086 8601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:52.249101 8601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:52.254719 8601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:52.255861 8601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:52.256916 8601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:52.267426 8606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:52.267442 8609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:52.267660 8601 server_base.cc:1034] running on GCE node
W20260928 23:58:52.267818 8607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:52.268541 8601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:52.269807 8601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:52.271006 8601 hybrid_clock.cc:648] HybridClock initialized: now 1790639932270995 us; error 28 us; skew 500 ppm
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:52.273981 8601 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:52.274889 8601 webserver.cc:460] Webserver started at http://127.0.0.1:40701/ using document root <none> and password file <none>
I20260928 23:58:52.275427 8601 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:52.275484 8601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:52.275658 8601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:52.277809 8601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "7a8f09590d7d4a1089fa1716d543edeb"
format_stamp: "Formatted at 2026-09-28 23:58:52 on dist-test-slave-wvpw"
server_key: "95077991f2fed3cb4b2db6c2e71f25d2"
server_key_iv: "dea0319668ded9068eecec60445bdbc7"
server_key_version: "encryptionkey@0"
I20260928 23:58:52.278373 8601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "7a8f09590d7d4a1089fa1716d543edeb"
format_stamp: "Formatted at 2026-09-28 23:58:52 on dist-test-slave-wvpw"
server_key: "95077991f2fed3cb4b2db6c2e71f25d2"
server_key_iv: "dea0319668ded9068eecec60445bdbc7"
server_key_version: "encryptionkey@0"
I20260928 23:58:52.282400 8601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:52.285097 8616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:52.286433 8601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:52.286587 8601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "7a8f09590d7d4a1089fa1716d543edeb"
format_stamp: "Formatted at 2026-09-28 23:58:52 on dist-test-slave-wvpw"
server_key: "95077991f2fed3cb4b2db6c2e71f25d2"
server_key_iv: "dea0319668ded9068eecec60445bdbc7"
server_key_version: "encryptionkey@0"
I20260928 23:58:52.286713 8601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:52.308048 8601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:52.311875 8601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:52.312098 8601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:52.320304 8601 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34381
I20260928 23:58:52.320333 8668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34381 every 8 connection(s)
I20260928 23:58:52.321484 8601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:52.324630 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:52.326032 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8601
I20260928 23:58:52.326179 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:52.326519 20832 external_mini_cluster.cc:1428] Setting key bf2d53bbd8d4f9e161079ce8cd350ff8
I20260928 23:58:52.331358 8669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb: Bootstrap starting.
I20260928 23:58:52.333875 8669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:52.334698 8669 log.cc:824] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:58:52.337232 8669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb: No bootstrap required, opened a new log
I20260928 23:58:52.341131 8669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260928 23:58:52.341482 8669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:52.341574 8669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8f09590d7d4a1089fa1716d543edeb, State: Initialized, Role: FOLLOWER
I20260928 23:58:52.342077 8669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260928 23:58:52.342209 8669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:52.342303 8669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:52.342384 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:52.328162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42579 (local address 127.0.0.1:34381)
0928 23:58:52.328692 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:52.328708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:52.328748 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:58:52.329461 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:52.329479 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:52.329849 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:52.330224 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:52.331088 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:52.332165 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:52.333142 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:52.333643 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:52.333824 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:52.337018 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:52.337024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:52.337040 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:52.337071 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:52.339000 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:52.339606 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:52.339612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:52.339617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:52.339733 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:52.340084 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:52.340090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:52.340096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:52.340589 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:52.340827 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:52.341151 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 23:58:52.341392 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":152,"threads_started":1}
I20260928 23:58:52.342407 8669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:52.343642 8669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260928 23:58:52.344035 8669 leader_election.cc:302] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a8f09590d7d4a1089fa1716d543edeb; no voters:
I20260928 23:58:52.344409 8669 leader_election.cc:288] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:52.344574 8674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:52.344841 8674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [term 1 LEADER]: Becoming Leader. State: Replica: 7a8f09590d7d4a1089fa1716d543edeb, State: Running, Role: LEADER
I20260928 23:58:52.345178 8674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260928 23:58:52.345755 8669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:52.345966 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:52.347012 8675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8f09590d7d4a1089fa1716d543edeb. Latest consensus state: current_term: 1 leader_uuid: "7a8f09590d7d4a1089fa1716d543edeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260928 23:58:52.347149 8675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:52.347438 8676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a8f09590d7d4a1089fa1716d543edeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8f09590d7d4a1089fa1716d543edeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260928 23:58:52.347549 8676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:52.347656 8684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:52.351202 8684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:52.357755 8684 catalog_manager.cc:1347] Generated new cluster ID: 1164491bef344f07898c3d6dd53f24e2
I20260928 23:58:52.357837 8684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:52.360291 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:52.348465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52183 (local address 127.0.0.1:34381)
0928 23:58:52.348570 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:52.348575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:52.351661 (+ 3086us) server_negotiation.cc:407] Connection header received
0928 23:58:52.351814 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:52.351817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:52.351869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:52.351968 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:52.352956 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:52.353575 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:52.354442 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:52.354679 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:52.354795 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:52.356342 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:52.356346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:52.356350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:52.356377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:52.358236 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:52.358857 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:52.358862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:52.358864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:52.358954 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:52.359308 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:52.359311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:52.359313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:52.359537 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:52.359706 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:52.359981 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 23:58:52.360088 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:58:52.393771 8684 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:52.395167 8684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:52.405820 8684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a8f09590d7d4a1089fa1716d543edeb: Generated new TSK 0
I20260928 23:58:52.406788 8684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:52.416529 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8601
W20260928 23:58:52.422606 8586 connection.cc:535] client connection to 127.0.0.1:34381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34381): Connection reset by peer (error 104)
2026-09-28T23:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8700](info): set up 2 sockets
Sep 28 23:58:52 dist-test-slave-wvpw krb5kdc[8700](info): commencing operation
krb5kdc: starting...
W20260928 23:58:53.464999 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:53 dist-test-slave-wvpw krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:53.600816 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:53.719421 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:53.719728 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:53.719774 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:53.723852 8713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:53.723925 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:53.723945 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:53.723963 8713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:53.723979 8713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:53.729557 8713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:53.730674 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:53.731709 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:53.741571 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:53.741557 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:53.741628 8713 server_base.cc:1034] running on GCE node
W20260928 23:58:53.741557 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:53.742538 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:53.743854 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:53.745091 8713 hybrid_clock.cc:648] HybridClock initialized: now 1790639933745034 us; error 78 us; skew 500 ppm
Sep 28 23:58:53 dist-test-slave-wvpw krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:53.748342 8713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:53.749426 8713 webserver.cc:460] Webserver started at http://127.0.0.1:41119/ using document root <none> and password file <none>
I20260928 23:58:53.750056 8713 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:53.750154 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:53.750396 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:53.752655 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "20d04b17d6b04770a3ead03185b4e8b8"
format_stamp: "Formatted at 2026-09-28 23:58:53 on dist-test-slave-wvpw"
server_key: "493efc81b1919ea99ce568cc35a1b8b5"
server_key_iv: "88145407a0c0266e86c8473098f149aa"
server_key_version: "encryptionkey@0"
I20260928 23:58:53.753261 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "20d04b17d6b04770a3ead03185b4e8b8"
format_stamp: "Formatted at 2026-09-28 23:58:53 on dist-test-slave-wvpw"
server_key: "493efc81b1919ea99ce568cc35a1b8b5"
server_key_iv: "88145407a0c0266e86c8473098f149aa"
server_key_version: "encryptionkey@0"
I20260928 23:58:53.757432 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:58:53.760048 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:53.761445 8713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:58:53.761605 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "20d04b17d6b04770a3ead03185b4e8b8"
format_stamp: "Formatted at 2026-09-28 23:58:53 on dist-test-slave-wvpw"
server_key: "493efc81b1919ea99ce568cc35a1b8b5"
server_key_iv: "88145407a0c0266e86c8473098f149aa"
server_key_version: "encryptionkey@0"
I20260928 23:58:53.761737 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:53.784524 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:53.787746 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:53.788009 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:53.795684 8713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40663
I20260928 23:58:53.795715 8780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40663 every 8 connection(s)
I20260928 23:58:53.796828 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:53.798388 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8713
I20260928 23:58:53.798540 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:53.798889 20832 external_mini_cluster.cc:1428] Setting key 6314d6ab9bbbb483b6cf42e61f8b929f
I20260928 23:58:53.800763 8781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:53.807283 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8: Bootstrap starting.
I20260928 23:58:53.808713 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:53.800583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34317 (local address 127.0.0.1:40663)
0928 23:58:53.801120 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:53.801141 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:53.801180 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:58:53.801882 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:53.801899 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:53.802256 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:53.802614 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:53.803076 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:53.804206 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:53.805165 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:53.805649 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:53.805819 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:53.806868 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:53.806873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:53.806886 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:53.807275 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:53.807328 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:53.807350 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:53.807497 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:53.807728 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 23:58:53.807875 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":169,"threads_started":1}
I20260928 23:58:53.810180 8781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:53.811105 8781 log.cc:824] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:53.811970 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:53.813723 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8: No bootstrap required, opened a new log
I20260928 23:58:53.819121 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:53.814418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42845 (local address 127.0.0.1:40663)
0928 23:58:53.814549 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:53.814554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:53.814735 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:58:53.814887 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:53.814889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:53.814926 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:53.814989 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:53.815312 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:53.815828 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:53.817179 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:53.817399 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:53.817482 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:53.818448 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:53.818451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:53.818453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:53.818468 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:53.818510 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:53.818523 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:53.818620 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:53.818890 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 23:58:53.818942 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:58:53.819265 8781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260928 23:58:53.819624 8781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:53.819695 8781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d04b17d6b04770a3ead03185b4e8b8, State: Initialized, Role: FOLLOWER
I20260928 23:58:53.820169 8781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260928 23:58:53.820421 8781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:53.820526 8781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:53.820642 8781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:53.821768 8781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260928 23:58:53.822227 8781 leader_election.cc:302] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20d04b17d6b04770a3ead03185b4e8b8; no voters:
I20260928 23:58:53.822621 8781 leader_election.cc:288] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:53.822751 8791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:53.823009 8791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 20d04b17d6b04770a3ead03185b4e8b8, State: Running, Role: LEADER
I20260928 23:58:53.823352 8791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260928 23:58:53.824112 8781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:53.825191 8792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20d04b17d6b04770a3ead03185b4e8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260928 23:58:53.825344 8792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:53.825625 8793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20d04b17d6b04770a3ead03185b4e8b8. Latest consensus state: current_term: 1 leader_uuid: "20d04b17d6b04770a3ead03185b4e8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d04b17d6b04770a3ead03185b4e8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260928 23:58:53.825712 8793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:53.825860 8800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:53.829569 8800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:53.835948 8800 catalog_manager.cc:1347] Generated new cluster ID: c530c477830c4e03b89656be0e0e90e9
I20260928 23:58:53.836050 8800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:53.852447 8800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:53.853494 8800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:53.864146 8800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20d04b17d6b04770a3ead03185b4e8b8: Generated new TSK 0
I20260928 23:58:53.865166 8800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:53.871475 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8713
W20260928 23:58:53.876994 8698 connection.cc:535] client connection to 127.0.0.1:40663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40663): Connection reset by peer (error 104)
2026-09-28T23:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:53 dist-test-slave-wvpw krb5kdc[8812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:53 dist-test-slave-wvpw krb5kdc[8812](info): set up 2 sockets
Sep 28 23:58:53 dist-test-slave-wvpw krb5kdc[8812](info): commencing operation
krb5kdc: starting...
W20260928 23:58:54.929033 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:55 dist-test-slave-wvpw krb5kdc[8812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:55.063683 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:55.181523 8825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:55.181833 8825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:55.181891 8825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:55.186151 8825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:55.186236 8825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:55.186257 8825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:55.186275 8825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:55.186290 8825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:55.192112 8825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:55.193363 8825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:55.194459 8825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:55.203387 8831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:55.203387 8833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:55.203403 8830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:55.203627 8825 server_base.cc:1034] running on GCE node
I20260928 23:58:55.204313 8825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:55.205561 8825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:55.206763 8825 hybrid_clock.cc:648] HybridClock initialized: now 1790639935206745 us; error 32 us; skew 500 ppm
Sep 28 23:58:55 dist-test-slave-wvpw krb5kdc[8812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:55.210022 8825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:55.211032 8825 webserver.cc:460] Webserver started at http://127.0.0.1:42113/ using document root <none> and password file <none>
I20260928 23:58:55.211635 8825 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:55.211731 8825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:55.211977 8825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:55.214165 8825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "ff6e32fab11a41f9a1a583aeda725b2a"
format_stamp: "Formatted at 2026-09-28 23:58:55 on dist-test-slave-wvpw"
server_key: "f2ebd0dd651cbe59b9a736a61166677b"
server_key_iv: "c81f3272785ad0e2b90979093f5deab8"
server_key_version: "encryptionkey@0"
I20260928 23:58:55.214775 8825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "ff6e32fab11a41f9a1a583aeda725b2a"
format_stamp: "Formatted at 2026-09-28 23:58:55 on dist-test-slave-wvpw"
server_key: "f2ebd0dd651cbe59b9a736a61166677b"
server_key_iv: "c81f3272785ad0e2b90979093f5deab8"
server_key_version: "encryptionkey@0"
I20260928 23:58:55.218772 8825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:58:55.221436 8840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:55.222782 8825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:55.222949 8825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "ff6e32fab11a41f9a1a583aeda725b2a"
format_stamp: "Formatted at 2026-09-28 23:58:55 on dist-test-slave-wvpw"
server_key: "f2ebd0dd651cbe59b9a736a61166677b"
server_key_iv: "c81f3272785ad0e2b90979093f5deab8"
server_key_version: "encryptionkey@0"
I20260928 23:58:55.223078 8825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:55.227852 8825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:55.243670 8825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:55.243912 8825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:55.251654 8825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40655
I20260928 23:58:55.251672 8892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40655 every 8 connection(s)
I20260928 23:58:55.252767 8825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:55.255764 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:55.261207 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8825
I20260928 23:58:55.261366 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:55.261633 20832 external_mini_cluster.cc:1428] Setting key d8c1faf74f369473938d1c8c3b4c4d51
I20260928 23:58:55.262094 8893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a: Bootstrap starting.
I20260928 23:58:55.264669 8893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:55.265609 8893 log.cc:824] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:55.267488 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:55.263217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39249 (local address 127.0.0.1:40655)
0928 23:58:55.263706 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:55.263722 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:55.263758 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:58:55.264536 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:55.264556 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:55.264917 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:55.265220 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:55.266146 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:55.266152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:55.266169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:55.266182 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:55.266247 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:55.266289 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:55.266408 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:55.266563 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:58:55.266748 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":165,"threads_started":1}
I20260928 23:58:55.267927 8893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a: No bootstrap required, opened a new log
I20260928 23:58:55.270390 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:55.272451 8893 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260928 23:58:55.272758 8893 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:55.272822 8893 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6e32fab11a41f9a1a583aeda725b2a, State: Initialized, Role: FOLLOWER
I20260928 23:58:55.273368 8893 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260928 23:58:55.273558 8893 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:55.273643 8893 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:55.273766 8893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:55.274773 8893 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260928 23:58:55.275151 8893 leader_election.cc:302] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6e32fab11a41f9a1a583aeda725b2a; no voters:
I20260928 23:58:55.275478 8893 leader_election.cc:288] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:55.275705 8903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:55.276031 8903 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [term 1 LEADER]: Becoming Leader. State: Replica: ff6e32fab11a41f9a1a583aeda725b2a, State: Running, Role: LEADER
I20260928 23:58:55.276458 8903 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260928 23:58:55.277128 8893 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:55.278302 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6e32fab11a41f9a1a583aeda725b2a. Latest consensus state: current_term: 1 leader_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260928 23:58:55.278448 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:55.278697 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6e32fab11a41f9a1a583aeda725b2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260928 23:58:55.278802 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:55.278918 8912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:55.281670 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:55.273046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50641 (local address 127.0.0.1:40655)
0928 23:58:55.273187 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:55.273190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:55.279612 (+ 6422us) server_negotiation.cc:407] Connection header received
0928 23:58:55.279792 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:55.279796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:55.279838 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:55.279939 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:55.280916 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:55.280922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:55.280925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:55.280945 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:55.280991 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:55.281008 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:55.281121 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:55.281489 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 23:58:55.281539 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:58:55.282737 8912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:55.289619 8912 catalog_manager.cc:1347] Generated new cluster ID: 22f88ee88dca4f3ca8c563ec7e1bc4b1
I20260928 23:58:55.289712 8912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:55.302925 8912 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:55.303308 8912 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:55.331234 8912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6e32fab11a41f9a1a583aeda725b2a: Generated new TSK 0
I20260928 23:58:55.332082 8912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:55.336313 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8825
2026-09-28T23:58:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:55 dist-test-slave-wvpw krb5kdc[8924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:55 dist-test-slave-wvpw krb5kdc[8924](info): set up 2 sockets
Sep 28 23:58:55 dist-test-slave-wvpw krb5kdc[8924](info): commencing operation
krb5kdc: starting...
W20260928 23:58:56.396979 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[8924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:56.528820 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:56.645651 8937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:56.645982 8937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:56.646055 8937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:56.650238 8937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:56.650358 8937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:56.650398 8937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:56.650431 8937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:56.650468 8937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:56.656486 8937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:56.657653 8937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:56.658711 8937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:56.667711 8943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:56.667742 8945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:56.667738 8942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:56.668044 8937 server_base.cc:1034] running on GCE node
I20260928 23:58:56.668769 8937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:56.669942 8937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:56.671140 8937 hybrid_clock.cc:648] HybridClock initialized: now 1790639936671128 us; error 35 us; skew 500 ppm
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[8924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:56.673961 8937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:56.675019 8937 webserver.cc:460] Webserver started at http://127.0.0.1:36295/ using document root <none> and password file <none>
I20260928 23:58:56.675617 8937 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:56.675710 8937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:56.675951 8937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:56.678222 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "aa3ae20add8341e6bb29056e1fa28640"
format_stamp: "Formatted at 2026-09-28 23:58:56 on dist-test-slave-wvpw"
server_key: "f444ada64cb1fb7bc27caf43ed409135"
server_key_iv: "1ff3a7a569ebd7c222d85185f4709529"
server_key_version: "encryptionkey@0"
I20260928 23:58:56.678848 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "aa3ae20add8341e6bb29056e1fa28640"
format_stamp: "Formatted at 2026-09-28 23:58:56 on dist-test-slave-wvpw"
server_key: "f444ada64cb1fb7bc27caf43ed409135"
server_key_iv: "1ff3a7a569ebd7c222d85185f4709529"
server_key_version: "encryptionkey@0"
I20260928 23:58:56.682929 8937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 23:58:56.685748 8952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:56.687129 8937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:58:56.687290 8937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "aa3ae20add8341e6bb29056e1fa28640"
format_stamp: "Formatted at 2026-09-28 23:58:56 on dist-test-slave-wvpw"
server_key: "f444ada64cb1fb7bc27caf43ed409135"
server_key_iv: "1ff3a7a569ebd7c222d85185f4709529"
server_key_version: "encryptionkey@0"
I20260928 23:58:56.687430 8937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:56.709100 8937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:56.712731 8937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:56.712973 8937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:56.720490 8937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41265
I20260928 23:58:56.720510 9004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41265 every 8 connection(s)
I20260928 23:58:56.721527 8937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:56.724454 9005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:56.726060 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 8937
I20260928 23:58:56.726217 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:56.726508 20832 external_mini_cluster.cc:1428] Setting key de6e878c669bd151e8568569c76abb1f
I20260928 23:58:56.731540 9005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: Bootstrap starting.
I20260928 23:58:56.734552 9005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:56.735716 9005 log.cc:824] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[8924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:58:56.737972 9005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: No bootstrap required, opened a new log
I20260928 23:58:56.741811 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:56.727931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48115 (local address 127.0.0.1:41265)
0928 23:58:56.728401 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:56.728418 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:56.728455 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:58:56.729133 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:56.729151 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:56.729512 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:56.729828 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:56.730806 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:56.731871 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:56.733103 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:56.733598 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:56.733724 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:56.736918 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:56.736924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:56.736939 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:56.736980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:56.738792 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:56.739283 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:56.739288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:56.739293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:56.739411 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:56.739714 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:56.739724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:56.739726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:56.740141 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:56.740384 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:56.740688 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 23:58:56.741013 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260928 23:58:56.741823 9005 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260928 23:58:56.742281 9005 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:56.742383 9005 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3ae20add8341e6bb29056e1fa28640, State: Initialized, Role: FOLLOWER
I20260928 23:58:56.742862 9005 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260928 23:58:56.743023 9005 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:56.743099 9005 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:56.743234 9005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:56.744513 9005 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260928 23:58:56.744907 9005 leader_election.cc:302] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa3ae20add8341e6bb29056e1fa28640; no voters:
I20260928 23:58:56.745182 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:56.745289 9005 leader_election.cc:288] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:56.745393 9010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:56.745561 9010 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [term 1 LEADER]: Becoming Leader. State: Replica: aa3ae20add8341e6bb29056e1fa28640, State: Running, Role: LEADER
I20260928 23:58:56.745954 9010 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260928 23:58:56.746711 9005 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:56.747766 9011 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa3ae20add8341e6bb29056e1fa28640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } } }
I20260928 23:58:56.747911 9011 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:56.747928 9012 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3ae20add8341e6bb29056e1fa28640. Latest consensus state: current_term: 1 leader_uuid: "aa3ae20add8341e6bb29056e1fa28640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3ae20add8341e6bb29056e1fa28640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } } }
I20260928 23:58:56.748042 9012 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640 [sys.catalog]: This master's current role is: LEADER
W20260928 23:58:56.751739 9030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:58:56.752128 9030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:58:56.752238 9024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:56.752996 9024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:56.757055 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:56.747605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.0.0.1:41265)
0928 23:58:56.747827 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:56.747832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:56.748066 (+ 234us) server_negotiation.cc:407] Connection header received
0928 23:58:56.748303 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:56.748307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:56.748365 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:56.748464 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:56.749661 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:56.750248 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:56.751120 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:56.751376 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:56.751489 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:56.753315 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:56.753320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:56.753323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:58:56.753351 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:56.755177 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:56.755659 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:56.755663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:56.755665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:56.755757 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:58:56.756052 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:58:56.756055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:58:56.756057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:58:56.756358 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:56.756491 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:56.756748 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 23:58:56.756872 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:58:56.759984 9024 catalog_manager.cc:1347] Generated new cluster ID: e49fd967b0f44f67ac0762030bea567b
I20260928 23:58:56.760058 9024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:56.766490 9024 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:56.767526 9024 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:56.773665 9024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa3ae20add8341e6bb29056e1fa28640: Generated new TSK 0
I20260928 23:58:56.774680 9024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:56.784015 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 8937
2026-09-28T23:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[9036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[9036](info): set up 2 sockets
Sep 28 23:58:56 dist-test-slave-wvpw krb5kdc[9036](info): commencing operation
krb5kdc: starting...
W20260928 23:58:57.825217 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:57 dist-test-slave-wvpw krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:57.960196 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:58.075537 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:58.075912 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:58.075995 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:58.080219 9049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:58.080381 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:58.080423 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:58.080462 9049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:58.080498 9049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:58.086213 9049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:58.087428 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:58.088560 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:58.097824 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:58.097867 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:58.097877 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:58.098033 9049 server_base.cc:1034] running on GCE node
I20260928 23:58:58.098732 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:58.099948 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:58.101154 9049 hybrid_clock.cc:648] HybridClock initialized: now 1790639938101127 us; error 49 us; skew 500 ppm
Sep 28 23:58:58 dist-test-slave-wvpw krb5kdc[9036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:58.104339 9049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:58.105371 9049 webserver.cc:460] Webserver started at http://127.0.0.1:36335/ using document root <none> and password file <none>
I20260928 23:58:58.105964 9049 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:58.106058 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:58.106305 9049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:58.108575 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "4083217167274e899107b648e8446d55"
format_stamp: "Formatted at 2026-09-28 23:58:58 on dist-test-slave-wvpw"
server_key: "657dbc5fc9df403a9f370d1e7afeefca"
server_key_iv: "4ff18fcab5f6583abb031665552ba978"
server_key_version: "encryptionkey@0"
I20260928 23:58:58.109197 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "4083217167274e899107b648e8446d55"
format_stamp: "Formatted at 2026-09-28 23:58:58 on dist-test-slave-wvpw"
server_key: "657dbc5fc9df403a9f370d1e7afeefca"
server_key_iv: "4ff18fcab5f6583abb031665552ba978"
server_key_version: "encryptionkey@0"
I20260928 23:58:58.113324 9049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:58:58.115916 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:58.117565 9049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:58:58.117735 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "4083217167274e899107b648e8446d55"
format_stamp: "Formatted at 2026-09-28 23:58:58 on dist-test-slave-wvpw"
server_key: "657dbc5fc9df403a9f370d1e7afeefca"
server_key_iv: "4ff18fcab5f6583abb031665552ba978"
server_key_version: "encryptionkey@0"
I20260928 23:58:58.117879 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:58.140343 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:58.143648 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:58.143899 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:58.151741 9049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35457
I20260928 23:58:58.151754 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35457 every 8 connection(s)
I20260928 23:58:58.152892 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:58.155969 9117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:58.157554 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9049
I20260928 23:58:58.157732 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:58.158034 20832 external_mini_cluster.cc:1428] Setting key 4f579675e3f56a10b51d273450d4c5e0
I20260928 23:58:58.162840 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55: Bootstrap starting.
I20260928 23:58:58.165468 9117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:58.166323 9117 log.cc:824] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:58.167697 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:58.159603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53839 (local address 127.0.0.1:35457)
0928 23:58:58.160116 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:58.160133 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:58.160171 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:58:58.160921 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:58.160941 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:58.161301 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:58.161668 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:58.162138 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:58.163196 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:58.164155 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:58.164725 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:58.164922 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:58.165885 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:58.165890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:58.165905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:58.166318 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:58.166378 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:58.166402 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:58.166517 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:58.166655 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:58:58.166907 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":159,"threads_started":1}
I20260928 23:58:58.168654 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55: No bootstrap required, opened a new log
I20260928 23:58:58.170912 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:58.172993 9117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260928 23:58:58.173362 9117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:58.173439 9117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4083217167274e899107b648e8446d55, State: Initialized, Role: FOLLOWER
I20260928 23:58:58.173965 9117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260928 23:58:58.174127 9117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:58.174218 9117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:58.174342 9117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:58.175366 9117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260928 23:58:58.175812 9117 leader_election.cc:302] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4083217167274e899107b648e8446d55; no voters:
I20260928 23:58:58.176209 9117 leader_election.cc:288] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:58.176347 9127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:58.176617 9127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [term 1 LEADER]: Becoming Leader. State: Replica: 4083217167274e899107b648e8446d55, State: Running, Role: LEADER
I20260928 23:58:58.177016 9127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } }
I20260928 23:58:58.177721 9117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:58.177863 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:58.173124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54157 (local address 127.0.0.1:35457)
0928 23:58:58.173267 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:58.173274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:58.173458 (+ 184us) server_negotiation.cc:407] Connection header received
0928 23:58:58.173602 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:58.173605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:58.173647 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:58.173742 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:58.174037 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:58.174740 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:58.175585 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:58:58.175966 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:58:58.176071 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:58:58.177122 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:58.177128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:58.177133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:58.177183 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:58.177218 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:58.177233 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:58.177316 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:58.177432 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 23:58:58.177508 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:58:58.178855 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4083217167274e899107b648e8446d55. Latest consensus state: current_term: 1 leader_uuid: "4083217167274e899107b648e8446d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } } }
I20260928 23:58:58.179000 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:58.179410 9136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:58.179437 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4083217167274e899107b648e8446d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4083217167274e899107b648e8446d55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35457 } } }
I20260928 23:58:58.179589 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:58.182181 9136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:58.188447 9136 catalog_manager.cc:1347] Generated new cluster ID: ffd07cef847546ac9182421466c3d86b
I20260928 23:58:58.188630 9136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:58.199847 9136 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:58.200902 9136 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:58.205794 9136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4083217167274e899107b648e8446d55: Generated new TSK 0
I20260928 23:58:58.206534 9136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:58.215235 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9049
2026-09-28T23:58:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:58 dist-test-slave-wvpw krb5kdc[9148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:58 dist-test-slave-wvpw krb5kdc[9148](info): set up 2 sockets
Sep 28 23:58:58 dist-test-slave-wvpw krb5kdc[9148](info): commencing operation
krb5kdc: starting...
W20260928 23:58:59.277534 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:58:59 dist-test-slave-wvpw krb5kdc[9148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:58:59.411624 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:58:59.525697 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:58:59.526072 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:58:59.526156 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:58:59.530411 9161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:58:59.530545 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:58:59.530587 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:58:59.530627 9161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:58:59.530663 9161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:58:59.536425 9161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:58:59.537675 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:58:59.538790 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:58:59.548071 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:59.548102 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:58:59.548064 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:58:59.548203 9161 server_base.cc:1034] running on GCE node
I20260928 23:58:59.549104 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:58:59.550374 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:58:59.551575 9161 hybrid_clock.cc:648] HybridClock initialized: now 1790639939551521 us; error 75 us; skew 500 ppm
Sep 28 23:58:59 dist-test-slave-wvpw krb5kdc[9148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:58:59.554626 9161 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:58:59.555589 9161 webserver.cc:460] Webserver started at http://127.0.0.1:37475/ using document root <none> and password file <none>
I20260928 23:58:59.556303 9161 fs_manager.cc:359] Metadata directory not provided
I20260928 23:58:59.556423 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:58:59.556676 9161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:58:59.558974 9161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "0e5779c44bd54b72b5fc8ae00b2c5977"
format_stamp: "Formatted at 2026-09-28 23:58:59 on dist-test-slave-wvpw"
server_key: "6b8772ab362361dfda011f355694f1bb"
server_key_iv: "92ebedd04ad319beef275dd338c098f2"
server_key_version: "encryptionkey@0"
I20260928 23:58:59.559577 9161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "0e5779c44bd54b72b5fc8ae00b2c5977"
format_stamp: "Formatted at 2026-09-28 23:58:59 on dist-test-slave-wvpw"
server_key: "6b8772ab362361dfda011f355694f1bb"
server_key_iv: "92ebedd04ad319beef275dd338c098f2"
server_key_version: "encryptionkey@0"
I20260928 23:58:59.563650 9161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:58:59.566419 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:58:59.567763 9161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:58:59.567942 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "0e5779c44bd54b72b5fc8ae00b2c5977"
format_stamp: "Formatted at 2026-09-28 23:58:59 on dist-test-slave-wvpw"
server_key: "6b8772ab362361dfda011f355694f1bb"
server_key_iv: "92ebedd04ad319beef275dd338c098f2"
server_key_version: "encryptionkey@0"
I20260928 23:58:59.568074 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:58:59.573058 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:58:59.576799 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:58:59.576973 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:58:59.584376 9161 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42651
I20260928 23:58:59.584436 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42651 every 8 connection(s)
I20260928 23:58:59.585430 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:58:59.588703 9229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:58:59.588711 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9161
I20260928 23:58:59.588905 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:58:59.589224 20832 external_mini_cluster.cc:1428] Setting key 41ad58811c094bf5f02b351f7cbedb91
I20260928 23:58:59.595175 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:59.590845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51661 (local address 127.0.0.1:42651)
0928 23:58:59.591330 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:59.591345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:59.591395 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:58:59.592345 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:59.592376 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:59.592682 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:59.593039 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:59.593913 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:59.593918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:59.593930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:59.593941 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:59.593996 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:59.594016 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:59.594172 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:59.594243 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:58:59.594424 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260928 23:58:59.596593 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: Bootstrap starting.
I20260928 23:58:59.598515 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:58:59.599416 9229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: Neither blocks nor log segments found. Creating new log.
I20260928 23:58:59.600478 9229 log.cc:824] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: Log is configured to *not* fsync() on all Append() calls
I20260928 23:58:59.602418 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:58:59.600704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36127 (local address 127.0.0.1:42651)
0928 23:58:59.600845 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:58:59.600851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:58:59.601078 (+ 227us) server_negotiation.cc:407] Connection header received
0928 23:58:59.601168 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:58:59.601173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:58:59.601212 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:58:59.601309 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:58:59.602022 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:58:59.602026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:58:59.602028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:58:59.602040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:58:59.602074 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:58:59.602085 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:58:59.602173 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:58:59.602254 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:58:59.602295 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:58:59.602980 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: No bootstrap required, opened a new log
I20260928 23:58:59.606987 9229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } }
I20260928 23:58:59.607247 9229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:58:59.607312 9229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5779c44bd54b72b5fc8ae00b2c5977, State: Initialized, Role: FOLLOWER
I20260928 23:58:59.607754 9229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } }
I20260928 23:58:59.607913 9229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:58:59.607985 9229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:58:59.608060 9229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:58:59.609087 9229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } }
I20260928 23:58:59.609464 9229 leader_election.cc:302] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5779c44bd54b72b5fc8ae00b2c5977; no voters:
I20260928 23:58:59.609838 9229 leader_election.cc:288] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:58:59.609995 9239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:58:59.610304 9239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5779c44bd54b72b5fc8ae00b2c5977, State: Running, Role: LEADER
I20260928 23:58:59.610642 9239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } }
I20260928 23:58:59.611195 9229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:58:59.612534 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5779c44bd54b72b5fc8ae00b2c5977. Latest consensus state: current_term: 1 leader_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } } }
I20260928 23:58:59.612684 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [sys.catalog]: This master's current role is: LEADER
I20260928 23:58:59.613088 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5779c44bd54b72b5fc8ae00b2c5977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42651 } } }
I20260928 23:58:59.613209 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977 [sys.catalog]: This master's current role is: LEADER
W20260928 23:58:59.616217 9254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:58:59.616391 9254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:58:59.616505 9250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:58:59.617542 9250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:58:59.624285 9250 catalog_manager.cc:1347] Generated new cluster ID: 71bc085f32834afb8fa44a50a9c12e57
I20260928 23:58:59.624380 9250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:58:59.652783 9250 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:58:59.653173 9250 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:58:59.661525 9250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e5779c44bd54b72b5fc8ae00b2c5977: Generated new TSK 0
I20260928 23:58:59.662393 9250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:58:59.674861 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9161
2026-09-28T23:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:58:59 dist-test-slave-wvpw krb5kdc[9260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:58:59 dist-test-slave-wvpw krb5kdc[9260](info): set up 2 sockets
Sep 28 23:58:59 dist-test-slave-wvpw krb5kdc[9260](info): commencing operation
krb5kdc: starting...
W20260928 23:59:00.725651 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:00 dist-test-slave-wvpw krb5kdc[9260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:00.860321 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:00.975385 9273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:00.975783 9273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:00.975879 9273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:00.980146 9273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:00.980309 9273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:00.980365 9273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:00.980430 9273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:00.980471 9273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:00.986253 9273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:00.987480 9273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:00.988612 9273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:00.997982 9278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:00.998116 9279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:00.997982 9281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:00.998625 9273 server_base.cc:1034] running on GCE node
I20260928 23:59:00.999358 9273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:01.000710 9273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:01.001952 9273 hybrid_clock.cc:648] HybridClock initialized: now 1790639941001925 us; error 43 us; skew 500 ppm
Sep 28 23:59:01 dist-test-slave-wvpw krb5kdc[9260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:01.012481 9273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:01.013566 9273 webserver.cc:460] Webserver started at http://127.0.0.1:41235/ using document root <none> and password file <none>
I20260928 23:59:01.014139 9273 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:01.014209 9273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:01.014403 9273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:01.016675 9273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "78c87c92e6cc477eae26816eb0d62b69"
format_stamp: "Formatted at 2026-09-28 23:59:01 on dist-test-slave-wvpw"
server_key: "f3128ebc63d303f86ecfbb8544e86c9e"
server_key_iv: "1f7f0770a2eb371fb4f51190f4e1d136"
server_key_version: "encryptionkey@0"
I20260928 23:59:01.017225 9273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "78c87c92e6cc477eae26816eb0d62b69"
format_stamp: "Formatted at 2026-09-28 23:59:01 on dist-test-slave-wvpw"
server_key: "f3128ebc63d303f86ecfbb8544e86c9e"
server_key_iv: "1f7f0770a2eb371fb4f51190f4e1d136"
server_key_version: "encryptionkey@0"
I20260928 23:59:01.021229 9273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:59:01.023839 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:01.025312 9273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:59:01.025460 9273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "78c87c92e6cc477eae26816eb0d62b69"
format_stamp: "Formatted at 2026-09-28 23:59:01 on dist-test-slave-wvpw"
server_key: "f3128ebc63d303f86ecfbb8544e86c9e"
server_key_iv: "1f7f0770a2eb371fb4f51190f4e1d136"
server_key_version: "encryptionkey@0"
I20260928 23:59:01.025584 9273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:01.046101 9273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:01.049787 9273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:01.049998 9273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:01.057550 9273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42327
I20260928 23:59:01.057572 9340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42327 every 8 connection(s)
I20260928 23:59:01.058585 9273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:01.061740 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:01.067651 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9273
I20260928 23:59:01.067862 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:01.068145 20832 external_mini_cluster.cc:1428] Setting key d938a49649f929d244e591af6ec246b4
I20260928 23:59:01.068156 9341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69: Bootstrap starting.
I20260928 23:59:01.070775 9341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:01.071595 9341 log.cc:824] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:01.073895 9341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69: No bootstrap required, opened a new log
Sep 28 23:59:01 dist-test-slave-wvpw krb5kdc[9260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:59:01.078810 9341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } }
I20260928 23:59:01.079118 9341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:01.079205 9341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c87c92e6cc477eae26816eb0d62b69, State: Initialized, Role: FOLLOWER
I20260928 23:59:01.079746 9341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } }
I20260928 23:59:01.079908 9341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:01.080008 9341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:01.080118 9341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:01.081125 9341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } }
I20260928 23:59:01.081508 9341 leader_election.cc:302] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c87c92e6cc477eae26816eb0d62b69; no voters:
I20260928 23:59:01.081897 9341 leader_election.cc:288] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:01.081969 9346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:01.082182 9346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [term 1 LEADER]: Becoming Leader. State: Replica: 78c87c92e6cc477eae26816eb0d62b69, State: Running, Role: LEADER
I20260928 23:59:01.082527 9346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } }
I20260928 23:59:01.083300 9341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:01.083786 9346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c87c92e6cc477eae26816eb0d62b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } } }
I20260928 23:59:01.083904 9346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:01.084116 9346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c87c92e6cc477eae26816eb0d62b69. Latest consensus state: current_term: 1 leader_uuid: "78c87c92e6cc477eae26816eb0d62b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c87c92e6cc477eae26816eb0d62b69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42327 } } }
I20260928 23:59:01.084218 9346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:01.084434 9350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:01.085068 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:01.069620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:48319 (local address 127.0.0.1:42327)
0928 23:59:01.070116 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:01.070132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:01.070175 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:59:01.070839 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:01.070856 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:01.071282 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:01.071619 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:01.072648 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:01.073661 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:01.074885 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:01.075385 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:01.075527 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:01.079111 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:01.079141 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:01.079162 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:01.079206 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:01.081237 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:01.081901 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:01.081908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:01.081915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:01.082099 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:01.082473 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:01.082477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:01.082479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:01.082975 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:01.083226 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:01.083517 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 23:59:01.083784 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":159,"threads_started":1}
I20260928 23:59:01.088624 9350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:01.089120 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:01.096496 9350 catalog_manager.cc:1347] Generated new cluster ID: 7a269a80cb9a45b287cad853605810fa
I20260928 23:59:01.096635 9350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:01.101353 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:01.091220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:33645 (local address 127.0.0.1:42327)
0928 23:59:01.091334 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:01.091339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:01.091735 (+ 396us) server_negotiation.cc:407] Connection header received
0928 23:59:01.091778 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:01.091781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:01.091829 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:01.091904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:01.092917 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:01.093673 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:01.095110 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:01.095484 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:01.095596 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:01.097566 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:01.097570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:01.097573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:01.097601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:01.099331 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:01.099955 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:01.099958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:01.099961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:01.100047 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:01.100437 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:01.100440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:01.100442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:01.100644 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:01.100783 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:01.101084 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:59:01.101181 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:59:01.106066 9350 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:01.107174 9350 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:01.112653 9350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78c87c92e6cc477eae26816eb0d62b69: Generated new TSK 0
I20260928 23:59:01.113456 9350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:01.127445 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9273
W20260928 23:59:01.133080 9258 connection.cc:535] client connection to 127.0.0.1:42327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42327): Connection reset by peer (error 104)
2026-09-28T23:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:01 dist-test-slave-wvpw krb5kdc[9372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:01 dist-test-slave-wvpw krb5kdc[9372](info): set up 2 sockets
Sep 28 23:59:01 dist-test-slave-wvpw krb5kdc[9372](info): commencing operation
krb5kdc: starting...
W20260928 23:59:02.197227 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:02 dist-test-slave-wvpw krb5kdc[9372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:02.332068 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:02.446280 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:02.446606 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:02.446661 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:02.451045 9385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:02.451128 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:02.451148 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:02.451165 9385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:02.451181 9385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:02.456897 9385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:02.458031 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:02.459035 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:02.468518 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:02.468518 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:02.468526 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:02.468573 9385 server_base.cc:1034] running on GCE node
I20260928 23:59:02.469426 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:02.470736 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:02.472127 9385 hybrid_clock.cc:648] HybridClock initialized: now 1790639942472092 us; error 48 us; skew 500 ppm
Sep 28 23:59:02 dist-test-slave-wvpw krb5kdc[9372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:02.475085 9385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:02.476332 9385 webserver.cc:460] Webserver started at http://127.0.0.1:36305/ using document root <none> and password file <none>
I20260928 23:59:02.476980 9385 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:02.477087 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:02.477350 9385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:02.479873 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "8bcd7fa741f54b258daeb78879b88951"
format_stamp: "Formatted at 2026-09-28 23:59:02 on dist-test-slave-wvpw"
server_key: "6bf42f695f328f3f5b80f62040a39ec4"
server_key_iv: "88c109e9b8cd64e38e2c7ed36b364a1c"
server_key_version: "encryptionkey@0"
I20260928 23:59:02.480587 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "8bcd7fa741f54b258daeb78879b88951"
format_stamp: "Formatted at 2026-09-28 23:59:02 on dist-test-slave-wvpw"
server_key: "6bf42f695f328f3f5b80f62040a39ec4"
server_key_iv: "88c109e9b8cd64e38e2c7ed36b364a1c"
server_key_version: "encryptionkey@0"
I20260928 23:59:02.484957 9385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:59:02.487752 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:02.489269 9385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:02.489451 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "8bcd7fa741f54b258daeb78879b88951"
format_stamp: "Formatted at 2026-09-28 23:59:02 on dist-test-slave-wvpw"
server_key: "6bf42f695f328f3f5b80f62040a39ec4"
server_key_iv: "88c109e9b8cd64e38e2c7ed36b364a1c"
server_key_version: "encryptionkey@0"
I20260928 23:59:02.489586 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:02.509243 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:02.512684 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:02.512920 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:02.520722 9385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39045
I20260928 23:59:02.520746 9452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39045 every 8 connection(s)
I20260928 23:59:02.521782 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:02.525154 9453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:02.529121 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9385
I20260928 23:59:02.529273 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:02.529567 20832 external_mini_cluster.cc:1428] Setting key 41de05437518a51571aadc0a6a89b4ee
I20260928 23:59:02.531689 9453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951: Bootstrap starting.
I20260928 23:59:02.534087 9453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:02.534898 9453 log.cc:824] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:02.537132 9453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951: No bootstrap required, opened a new log
I20260928 23:59:02.539470 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:02.531096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:40583 (local address 127.0.0.1:39045)
0928 23:59:02.531566 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:02.531582 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:02.531620 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:59:02.532427 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:02.532446 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:02.532812 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:02.533195 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:02.533632 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:02.534723 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:02.535721 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:02.536211 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:02.536437 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:02.537491 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:02.537496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:02.537511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:02.537925 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:02.537983 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:02.538006 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:02.538159 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:02.538361 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:59:02.538509 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260928 23:59:02.540740 9453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } }
I20260928 23:59:02.541005 9453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:02.541087 9453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcd7fa741f54b258daeb78879b88951, State: Initialized, Role: FOLLOWER
I20260928 23:59:02.541555 9453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } }
I20260928 23:59:02.541730 9453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:02.541798 9453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:02.541926 9453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:02.542919 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:02.543155 9453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } }
I20260928 23:59:02.543515 9453 leader_election.cc:302] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bcd7fa741f54b258daeb78879b88951; no voters:
I20260928 23:59:02.543851 9453 leader_election.cc:288] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:02.543967 9458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:02.544210 9458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [term 1 LEADER]: Becoming Leader. State: Replica: 8bcd7fa741f54b258daeb78879b88951, State: Running, Role: LEADER
I20260928 23:59:02.544674 9458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } }
I20260928 23:59:02.545322 9453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:02.546380 9459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bcd7fa741f54b258daeb78879b88951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } } }
I20260928 23:59:02.546512 9459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:02.546806 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bcd7fa741f54b258daeb78879b88951. Latest consensus state: current_term: 1 leader_uuid: "8bcd7fa741f54b258daeb78879b88951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd7fa741f54b258daeb78879b88951" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39045 } } }
I20260928 23:59:02.546968 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:02.547298 9471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:02.550720 9471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:02.551842 9456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:02.546347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:57871 (local address 127.0.0.1:39045)
0928 23:59:02.546468 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:02.546473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:02.547091 (+ 618us) server_negotiation.cc:407] Connection header received
0928 23:59:02.547312 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:02.547318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:02.547378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:02.547435 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:02.547879 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:02.548708 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:02.549851 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:02.550188 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:02.550305 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:02.551299 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:02.551304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:02.551307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:02.551322 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:02.551353 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:02.551367 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:02.551466 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:02.551647 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 23:59:02.551690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:59:02.557260 9471 catalog_manager.cc:1347] Generated new cluster ID: 2647e9d03e9142cfbe1cb124368c86c6
I20260928 23:59:02.557348 9471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:02.579926 9471 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:02.581051 9471 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:02.590139 9471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bcd7fa741f54b258daeb78879b88951: Generated new TSK 0
I20260928 23:59:02.590971 9471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:02.598719 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9385
W20260928 23:59:02.604496 9370 connection.cc:535] client connection to 127.0.0.1:39045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39045): Connection reset by peer (error 104)
2026-09-28T23:59:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:02 dist-test-slave-wvpw krb5kdc[9484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:02 dist-test-slave-wvpw krb5kdc[9484](info): set up 2 sockets
Sep 28 23:59:02 dist-test-slave-wvpw krb5kdc[9484](info): commencing operation
krb5kdc: starting...
W20260928 23:59:03.657456 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:03 dist-test-slave-wvpw krb5kdc[9484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:03.800861 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:03.916081 9497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:03.916411 9497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:03.916512 9497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:03.920928 9497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:03.921008 9497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:03.921031 9497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:03.921048 9497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:03.921105 9497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:03.927155 9497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:03.928380 9497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:03.929493 9497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:03.938622 9505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:03.938622 9503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:03.938860 9497 server_base.cc:1034] running on GCE node
W20260928 23:59:03.938747 9502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:03.939522 9497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:03.940819 9497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:03.942008 9497 hybrid_clock.cc:648] HybridClock initialized: now 1790639943941993 us; error 29 us; skew 500 ppm
Sep 28 23:59:03 dist-test-slave-wvpw krb5kdc[9484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:03.944908 9497 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:03.945818 9497 webserver.cc:460] Webserver started at http://127.0.0.1:35495/ using document root <none> and password file <none>
I20260928 23:59:03.946359 9497 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:03.946421 9497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:03.946589 9497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:03.948812 9497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "310836c7cea84cbcbb7b7c1a6fda072f"
format_stamp: "Formatted at 2026-09-28 23:59:03 on dist-test-slave-wvpw"
server_key: "f57283b0485c66154733630898b58feb"
server_key_iv: "19bf36895f26fcf7acced509dd68ef08"
server_key_version: "encryptionkey@0"
I20260928 23:59:03.949368 9497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "310836c7cea84cbcbb7b7c1a6fda072f"
format_stamp: "Formatted at 2026-09-28 23:59:03 on dist-test-slave-wvpw"
server_key: "f57283b0485c66154733630898b58feb"
server_key_iv: "19bf36895f26fcf7acced509dd68ef08"
server_key_version: "encryptionkey@0"
I20260928 23:59:03.953400 9497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:59:03.956317 9512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:03.957865 9497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:03.958109 9497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "310836c7cea84cbcbb7b7c1a6fda072f"
format_stamp: "Formatted at 2026-09-28 23:59:03 on dist-test-slave-wvpw"
server_key: "f57283b0485c66154733630898b58feb"
server_key_iv: "19bf36895f26fcf7acced509dd68ef08"
server_key_version: "encryptionkey@0"
I20260928 23:59:03.958240 9497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:03.963183 9497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:03.966835 9497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:03.967088 9497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:03.975327 9497 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40463
I20260928 23:59:03.975411 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40463 every 8 connection(s)
I20260928 23:59:03.976564 9497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:03.977983 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9497
I20260928 23:59:03.978152 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:03.978452 20832 external_mini_cluster.cc:1428] Setting key df58a99a62764c3f6d194922b29fa5c1
I20260928 23:59:03.980024 9565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:03.984725 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:03.979962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:33373 (local address 127.0.0.1:40463)
0928 23:59:03.980459 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:03.980481 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:03.980528 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:59:03.981455 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:03.981479 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:03.981877 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:03.982299 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:03.983029 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:03.983036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:03.983051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:03.983062 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:03.983124 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:03.983155 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:03.983325 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:03.983476 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 23:59:03.983656 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":203,"threads_started":1}
I20260928 23:59:03.987798 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f: Bootstrap starting.
I20260928 23:59:03.988088 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:03.990527 9565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:03.991629 9565 log.cc:824] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:03.993444 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:03.990479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:49271 (local address 127.0.0.1:40463)
0928 23:59:03.990619 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:03.990623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:03.990805 (+ 182us) server_negotiation.cc:407] Connection header received
0928 23:59:03.990986 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:03.990994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:03.991031 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:03.991123 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:03.993059 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:03.993062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:03.993065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:03.993077 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:03.993106 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:03.993117 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:03.993216 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:03.993301 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 23:59:03.993339 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:59:03.994370 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f: No bootstrap required, opened a new log
I20260928 23:59:03.998387 9565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 23:59:03.998663 9565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:03.998700 9565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310836c7cea84cbcbb7b7c1a6fda072f, State: Initialized, Role: FOLLOWER
I20260928 23:59:03.999207 9565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 23:59:03.999364 9565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:03.999460 9565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:03.999567 9565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:04.000672 9565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 23:59:04.001049 9565 leader_election.cc:302] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310836c7cea84cbcbb7b7c1a6fda072f; no voters:
I20260928 23:59:04.001395 9565 leader_election.cc:288] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:04.001641 9575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:04.001952 9575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [term 1 LEADER]: Becoming Leader. State: Replica: 310836c7cea84cbcbb7b7c1a6fda072f, State: Running, Role: LEADER
I20260928 23:59:04.002338 9575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 23:59:04.002828 9565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:04.004309 9577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310836c7cea84cbcbb7b7c1a6fda072f. Latest consensus state: current_term: 1 leader_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 23:59:04.004338 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310836c7cea84cbcbb7b7c1a6fda072f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 23:59:04.004472 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:04.004472 9577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:04.004962 9587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:04.008327 9587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:04.014663 9587 catalog_manager.cc:1347] Generated new cluster ID: 41a62eb18538460c97cfd8f37ef85de2
I20260928 23:59:04.014783 9587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:04.040793 9587 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:04.041193 9587 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:04.047638 9587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310836c7cea84cbcbb7b7c1a6fda072f: Generated new TSK 0
I20260928 23:59:04.048465 9587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:04.052980 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9497
2026-09-28T23:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:04 dist-test-slave-wvpw krb5kdc[9596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:04 dist-test-slave-wvpw krb5kdc[9596](info): set up 2 sockets
Sep 28 23:59:04 dist-test-slave-wvpw krb5kdc[9596](info): commencing operation
krb5kdc: starting...
W20260928 23:59:05.121305 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:05.269848 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:05.389225 9609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:05.389590 9609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:05.389669 9609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:05.393935 9609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:05.394070 9609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:05.394110 9609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:05.394160 9609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:05.394199 9609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:05.399952 9609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:05.401245 9609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:05.402381 9609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:05.411830 9617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:05.411830 9614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:05.411854 9615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:05.412284 9609 server_base.cc:1034] running on GCE node
I20260928 23:59:05.412982 9609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:05.414220 9609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:05.415441 9609 hybrid_clock.cc:648] HybridClock initialized: now 1790639945415413 us; error 44 us; skew 500 ppm
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:05.418758 9609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:05.419806 9609 webserver.cc:460] Webserver started at http://127.0.0.1:42881/ using document root <none> and password file <none>
I20260928 23:59:05.420529 9609 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:05.420632 9609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:05.420881 9609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:05.423111 9609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "5fe99d862107402ea7f44c00185e4698"
format_stamp: "Formatted at 2026-09-28 23:59:05 on dist-test-slave-wvpw"
server_key: "f29937a4704d3cd54841e399ddd8ecab"
server_key_iv: "6432b420266ac5fe7fc18a4d6b76bd79"
server_key_version: "encryptionkey@0"
I20260928 23:59:05.423763 9609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "5fe99d862107402ea7f44c00185e4698"
format_stamp: "Formatted at 2026-09-28 23:59:05 on dist-test-slave-wvpw"
server_key: "f29937a4704d3cd54841e399ddd8ecab"
server_key_iv: "6432b420266ac5fe7fc18a4d6b76bd79"
server_key_version: "encryptionkey@0"
I20260928 23:59:05.427883 9609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:59:05.430800 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:05.432394 9609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:05.432559 9609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "5fe99d862107402ea7f44c00185e4698"
format_stamp: "Formatted at 2026-09-28 23:59:05 on dist-test-slave-wvpw"
server_key: "f29937a4704d3cd54841e399ddd8ecab"
server_key_iv: "6432b420266ac5fe7fc18a4d6b76bd79"
server_key_version: "encryptionkey@0"
I20260928 23:59:05.432699 9609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:05.459738 9609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:05.463557 9609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:05.463804 9609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:05.471537 9609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39267
I20260928 23:59:05.471552 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39267 every 8 connection(s)
I20260928 23:59:05.472759 9609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:05.475940 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:05.478019 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9609
I20260928 23:59:05.478191 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:05.478475 20832 external_mini_cluster.cc:1428] Setting key d8b31d8e5a6716ff626bc9b3f7f2c681
I20260928 23:59:05.483206 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: Bootstrap starting.
I20260928 23:59:05.485862 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:05.486668 9677 log.cc:824] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:59:05.488886 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: No bootstrap required, opened a new log
I20260928 23:59:05.492843 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } }
I20260928 23:59:05.493182 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:05.493266 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe99d862107402ea7f44c00185e4698, State: Initialized, Role: FOLLOWER
I20260928 23:59:05.493821 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:05.480105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:59123 (local address 127.0.0.1:39267)
0928 23:59:05.480616 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:05.480631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:05.480670 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:59:05.481423 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:05.481444 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:05.481838 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:05.482207 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:05.483072 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:05.484155 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:05.485164 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:05.485671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:05.485854 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:05.488615 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:05.488621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:05.488635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:05.488664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:05.490542 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:05.491121 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:05.491126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:05.491131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:05.491261 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:05.491614 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:05.491624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:05.491626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:05.492047 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:05.492324 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:05.492669 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 23:59:05.492903 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":140,"threads_started":1}
I20260928 23:59:05.493795 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } }
I20260928 23:59:05.493969 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:05.494050 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:05.494177 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:05.495249 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } }
I20260928 23:59:05.495625 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe99d862107402ea7f44c00185e4698; no voters:
I20260928 23:59:05.496002 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:05.496133 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:05.496385 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe99d862107402ea7f44c00185e4698, State: Running, Role: LEADER
I20260928 23:59:05.496771 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } }
I20260928 23:59:05.497175 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:05.497535 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:05.499645 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe99d862107402ea7f44c00185e4698. Latest consensus state: current_term: 1 leader_uuid: "5fe99d862107402ea7f44c00185e4698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } } }
I20260928 23:59:05.500072 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe99d862107402ea7f44c00185e4698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe99d862107402ea7f44c00185e4698" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39267 } } }
I20260928 23:59:05.500154 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:05.500165 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698 [sys.catalog]: This master's current role is: LEADER
W20260928 23:59:05.502588 9700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:05.502701 9700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:05.502777 9703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:05.503760 9703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:05.510443 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:05.499426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:55943 (local address 127.0.0.1:39267)
0928 23:59:05.499567 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:05.499573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:05.500631 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 23:59:05.500882 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:05.500885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:05.500942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:05.501030 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:05.501966 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:05.502894 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:05.504115 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:05.504466 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:05.504546 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:05.506570 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:05.506574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:05.506577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:05.506614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:05.508333 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:05.508859 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:05.508863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:05.508865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:05.508958 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:05.509391 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:05.509395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:05.509397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:05.509637 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:05.509739 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:05.510169 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 23:59:05.510287 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:59:05.512053 9703 catalog_manager.cc:1347] Generated new cluster ID: 0c7a0e5d5e2c4f19912421051848eb0d
I20260928 23:59:05.512130 9703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:05.521173 9703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:05.522282 9703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:05.532902 9703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe99d862107402ea7f44c00185e4698: Generated new TSK 0
I20260928 23:59:05.533704 9703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:05.538784 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9609
W20260928 23:59:05.544030 9594 connection.cc:535] client connection to 127.0.0.1:39267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39267): Connection reset by peer (error 104)
2026-09-28T23:59:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9708](info): set up 2 sockets
Sep 28 23:59:05 dist-test-slave-wvpw krb5kdc[9708](info): commencing operation
krb5kdc: starting...
W20260928 23:59:06.608906 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:06 dist-test-slave-wvpw krb5kdc[9708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:06.745383 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:06.862367 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:06.862682 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:06.862735 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:06.866961 9721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:06.867045 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:06.867065 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:06.867084 9721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:06.867108 9721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:06.872728 9721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:06.873831 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:06.874863 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:06.883975 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:06.883991 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:06.883991 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:06.884411 9721 server_base.cc:1034] running on GCE node
I20260928 23:59:06.885000 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:06.886216 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:06.887424 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790639946887407 us; error 42 us; skew 500 ppm
Sep 28 23:59:06 dist-test-slave-wvpw krb5kdc[9708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:06.899427 9721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:06.900591 9721 webserver.cc:460] Webserver started at http://127.0.0.1:41659/ using document root <none> and password file <none>
I20260928 23:59:06.901260 9721 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:06.901364 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:06.901615 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:06.903993 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "a66825c4c9dd490f83a1fe9855d2d07a"
format_stamp: "Formatted at 2026-09-28 23:59:06 on dist-test-slave-wvpw"
server_key: "1398174d6ab41af2f8583fdfe97fef4a"
server_key_iv: "ae2bbac01623293e4c5e68561e73063f"
server_key_version: "encryptionkey@0"
I20260928 23:59:06.904681 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "a66825c4c9dd490f83a1fe9855d2d07a"
format_stamp: "Formatted at 2026-09-28 23:59:06 on dist-test-slave-wvpw"
server_key: "1398174d6ab41af2f8583fdfe97fef4a"
server_key_iv: "ae2bbac01623293e4c5e68561e73063f"
server_key_version: "encryptionkey@0"
I20260928 23:59:06.908751 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:59:06.911512 9736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:06.913105 9721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:06.913268 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "a66825c4c9dd490f83a1fe9855d2d07a"
format_stamp: "Formatted at 2026-09-28 23:59:06 on dist-test-slave-wvpw"
server_key: "1398174d6ab41af2f8583fdfe97fef4a"
server_key_iv: "ae2bbac01623293e4c5e68561e73063f"
server_key_version: "encryptionkey@0"
I20260928 23:59:06.913401 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:06.945580 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:06.949869 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:06.950060 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:06.958475 9721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45951
I20260928 23:59:06.958534 9788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45951 every 8 connection(s)
I20260928 23:59:06.959635 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:06.962636 9789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:06.963513 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9721
I20260928 23:59:06.963725 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:06.964041 20832 external_mini_cluster.cc:1428] Setting key 39b23d67409e30d8d27215f5c355c560
I20260928 23:59:06.969741 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a: Bootstrap starting.
I20260928 23:59:06.972678 9789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:06.973583 9789 log.cc:824] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:06.974237 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:06.965547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:48085 (local address 127.0.0.1:45951)
0928 23:59:06.966019 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:06.966036 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:06.966074 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:59:06.966738 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:06.966756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:06.967129 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:06.967435 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:06.967860 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:06.969019 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:06.970315 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:06.970881 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:06.971014 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:06.972134 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:06.972154 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:06.972170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:06.972684 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:06.972778 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:06.972817 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:06.972981 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:06.973187 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 23:59:06.973342 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260928 23:59:06.976164 9789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a: No bootstrap required, opened a new log
I20260928 23:59:06.978129 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:06.980795 9789 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260928 23:59:06.981177 9789 raft_consensus.cc:374] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:06.981290 9789 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66825c4c9dd490f83a1fe9855d2d07a, State: Initialized, Role: FOLLOWER
I20260928 23:59:06.981787 9789 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260928 23:59:06.981963 9789 raft_consensus.cc:388] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:06.982056 9789 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:06.982188 9789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:06.983227 9789 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260928 23:59:06.983611 9789 leader_election.cc:302] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a66825c4c9dd490f83a1fe9855d2d07a; no voters:
I20260928 23:59:06.984375 9789 leader_election.cc:288] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:06.984380 9799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:06.984676 9799 raft_consensus.cc:686] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [term 1 LEADER]: Becoming Leader. State: Replica: a66825c4c9dd490f83a1fe9855d2d07a, State: Running, Role: LEADER
I20260928 23:59:06.985114 9799 consensus_queue.cc:237] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260928 23:59:06.985778 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:06.980832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:34423 (local address 127.0.0.1:45951)
0928 23:59:06.980972 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:06.980976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:06.981112 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:59:06.981251 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:06.981254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:06.981318 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:06.981431 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:06.981874 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:06.982598 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:06.983520 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:06.983886 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:06.984001 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:06.985187 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:06.985228 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:06.985231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:06.985248 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:06.985280 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:06.985296 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:06.985363 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:06.985596 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 23:59:06.985637 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:59:06.985901 9789 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:06.987628 9801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } } }
I20260928 23:59:06.987792 9801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:06.988124 9800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66825c4c9dd490f83a1fe9855d2d07a. Latest consensus state: current_term: 1 leader_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66825c4c9dd490f83a1fe9855d2d07a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } } }
I20260928 23:59:06.988224 9800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:06.988204 9813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:06.991677 9813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:06.998193 9813 catalog_manager.cc:1347] Generated new cluster ID: b545f5c0d1094db9b42a6fd14679b577
I20260928 23:59:06.998283 9813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:07.012099 9813 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:07.013290 9813 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:07.026459 9813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a66825c4c9dd490f83a1fe9855d2d07a: Generated new TSK 0
I20260928 23:59:07.027305 9813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:07.037510 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9721
W20260928 23:59:07.043141 9706 connection.cc:535] client connection to 127.0.0.1:45951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45951): Connection reset by peer (error 104)
2026-09-28T23:59:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:07 dist-test-slave-wvpw krb5kdc[9820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:07 dist-test-slave-wvpw krb5kdc[9820](info): set up 2 sockets
Sep 28 23:59:07 dist-test-slave-wvpw krb5kdc[9820](info): commencing operation
krb5kdc: starting...
W20260928 23:59:08.077086 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:08 dist-test-slave-wvpw krb5kdc[9820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:59:08.211599 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:08.330220 9833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:08.330575 9833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:08.330667 9833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:08.335098 9833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:08.335215 9833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:08.335254 9833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:08.335291 9833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:08.335327 9833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:08.341975 9833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:08.343184 9833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:08.344303 9833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:08.353327 9841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:08.353330 9838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:08.353360 9839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:08.353673 9833 server_base.cc:1034] running on GCE node
I20260928 23:59:08.354352 9833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:08.355444 9833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:08.356698 9833 hybrid_clock.cc:648] HybridClock initialized: now 1790639948356676 us; error 37 us; skew 500 ppm
Sep 28 23:59:08 dist-test-slave-wvpw krb5kdc[9820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:08.359673 9833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:59:08.360842 9833 webserver.cc:460] Webserver started at http://127.0.0.1:41879/ using document root <none> and password file <none>
I20260928 23:59:08.361445 9833 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:08.361539 9833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:08.361791 9833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:08.363955 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "c2ad3fb1607d491fb1bb18bc9469e394"
format_stamp: "Formatted at 2026-09-28 23:59:08 on dist-test-slave-wvpw"
server_key: "b314190a1bf19c1d158b30bf8456852f"
server_key_iv: "3ca483245e631caea19305627ccdf93e"
server_key_version: "encryptionkey@0"
I20260928 23:59:08.364598 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "c2ad3fb1607d491fb1bb18bc9469e394"
format_stamp: "Formatted at 2026-09-28 23:59:08 on dist-test-slave-wvpw"
server_key: "b314190a1bf19c1d158b30bf8456852f"
server_key_iv: "3ca483245e631caea19305627ccdf93e"
server_key_version: "encryptionkey@0"
I20260928 23:59:08.368567 9833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:59:08.371284 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:08.372886 9833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:08.373059 9833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "c2ad3fb1607d491fb1bb18bc9469e394"
format_stamp: "Formatted at 2026-09-28 23:59:08 on dist-test-slave-wvpw"
server_key: "b314190a1bf19c1d158b30bf8456852f"
server_key_iv: "3ca483245e631caea19305627ccdf93e"
server_key_version: "encryptionkey@0"
I20260928 23:59:08.373193 9833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:08.378132 9833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:08.381704 9833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:08.381935 9833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:08.389945 9833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39503
I20260928 23:59:08.389953 9900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39503 every 8 connection(s)
I20260928 23:59:08.390956 9833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:08.394054 9901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:08.399051 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9833
I20260928 23:59:08.399220 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:08.399518 20832 external_mini_cluster.cc:1428] Setting key 993e332031dbb6373fa11a95ae7caf05
I20260928 23:59:08.400384 9901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394: Bootstrap starting.
I20260928 23:59:08.402961 9901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:08.403970 9901 log.cc:824] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:08.405596 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:08.401103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:39991 (local address 127.0.0.1:39503)
0928 23:59:08.401607 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:08.401625 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:08.401665 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:59:08.402386 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:08.402406 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:08.402717 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:08.403107 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:08.404038 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:08.404044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:08.404057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:08.404132 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:08.404284 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:08.404330 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:08.404471 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:08.404612 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 23:59:08.404802 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":164,"threads_started":1}
I20260928 23:59:08.406250 9901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394: No bootstrap required, opened a new log
I20260928 23:59:08.408682 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:08.410751 9901 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260928 23:59:08.411232 9901 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:08.411453 9901 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ad3fb1607d491fb1bb18bc9469e394, State: Initialized, Role: FOLLOWER
I20260928 23:59:08.412285 9901 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260928 23:59:08.412485 9901 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:08.412563 9901 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:08.412714 9901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:08.413452 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:08.411486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:37465 (local address 127.0.0.1:39503)
0928 23:59:08.411740 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:08.411745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:08.411910 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:59:08.412045 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:08.412048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:08.412094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:08.412181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:08.413035 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:08.413038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:08.413041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:08.413053 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:08.413085 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:08.413101 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:08.413195 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:08.413283 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:59:08.413330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 23:59:08.414000 9901 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260928 23:59:08.414448 9901 leader_election.cc:302] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ad3fb1607d491fb1bb18bc9469e394; no voters:
I20260928 23:59:08.414839 9901 leader_election.cc:288] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:08.414950 9911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:08.415198 9911 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [term 1 LEADER]: Becoming Leader. State: Replica: c2ad3fb1607d491fb1bb18bc9469e394, State: Running, Role: LEADER
I20260928 23:59:08.415582 9911 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } }
I20260928 23:59:08.416278 9901 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:08.417506 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ad3fb1607d491fb1bb18bc9469e394. Latest consensus state: current_term: 1 leader_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } } }
I20260928 23:59:08.417506 9912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad3fb1607d491fb1bb18bc9469e394" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39503 } } }
I20260928 23:59:08.417768 9912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:08.417794 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:08.418715 9921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:08.422385 9921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:08.428503 9921 catalog_manager.cc:1347] Generated new cluster ID: 75d400efe5464f7fa25c3099546ddc37
I20260928 23:59:08.428593 9921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:08.444670 9921 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:08.445057 9921 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:08.454746 9921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ad3fb1607d491fb1bb18bc9469e394: Generated new TSK 0
I20260928 23:59:08.455693 9921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:08.465337 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9833
2026-09-28T23:59:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:08 dist-test-slave-wvpw krb5kdc[9932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:08 dist-test-slave-wvpw krb5kdc[9932](info): set up 2 sockets
Sep 28 23:59:08 dist-test-slave-wvpw krb5kdc[9932](info): commencing operation
krb5kdc: starting...
W20260928 23:59:09.509097 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:09.640348 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:40953
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:09.758585 9945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:09.758913 9945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:09.758958 9945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:09.763314 9945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:09.763408 9945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:09.763429 9945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:09.763447 9945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:09.763463 9945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:09.769395 9945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:40953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:09.770632 9945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:09.771835 9945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:09.781066 9951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:09.780978 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:09.781316 9945 server_base.cc:1034] running on GCE node
W20260928 23:59:09.781371 9950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:09.782017 9945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:09.783236 9945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:09.784431 9945 hybrid_clock.cc:648] HybridClock initialized: now 1790639949784417 us; error 28 us; skew 500 ppm
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[9932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:09.787341 9945 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:09.788462 9945 webserver.cc:460] Webserver started at http://127.20.88.62:34773/ using document root <none> and password file <none>
I20260928 23:59:09.789057 9945 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:09.789152 9945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:09.789390 9945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:09.791514 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "bc9d461c23ea408f9d19261a4313ebed"
format_stamp: "Formatted at 2026-09-28 23:59:09 on dist-test-slave-wvpw"
server_key: "34c7112f22871cc269f161e13b25f5ea"
server_key_iv: "614e28f20cf6ab06ab0e3a761f36b8c4"
server_key_version: "encryptionkey@0"
I20260928 23:59:09.792127 9945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "bc9d461c23ea408f9d19261a4313ebed"
format_stamp: "Formatted at 2026-09-28 23:59:09 on dist-test-slave-wvpw"
server_key: "34c7112f22871cc269f161e13b25f5ea"
server_key_iv: "614e28f20cf6ab06ab0e3a761f36b8c4"
server_key_version: "encryptionkey@0"
I20260928 23:59:09.796070 9945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:59:09.798760 9960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:09.800170 9945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:59:09.800367 9945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "bc9d461c23ea408f9d19261a4313ebed"
format_stamp: "Formatted at 2026-09-28 23:59:09 on dist-test-slave-wvpw"
server_key: "34c7112f22871cc269f161e13b25f5ea"
server_key_iv: "614e28f20cf6ab06ab0e3a761f36b8c4"
server_key_version: "encryptionkey@0"
I20260928 23:59:09.800503 9945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:09.812608 9945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:09.816125 9945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:09.816376 9945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:09.823720 9945 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:40953
I20260928 23:59:09.823733 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:40953 every 8 connection(s)
I20260928 23:59:09.824797 9945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:09.827463 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 9945
I20260928 23:59:09.827701 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:09.827965 10013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:09.828028 20832 external_mini_cluster.cc:1428] Setting key 1eed3b0508ad36e843db4bcb110fdfc0
I20260928 23:59:09.835120 10013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed: Bootstrap starting.
I20260928 23:59:09.837743 10013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed: Neither blocks nor log segments found. Creating new log.
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[9932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:09.838651 10013 log.cc:824] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:09.841404 10013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed: No bootstrap required, opened a new log
I20260928 23:59:09.844193 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:09.829534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41895 (local address 127.20.88.62:40953)
0928 23:59:09.830044 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:09.830061 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:09.830098 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:59:09.831046 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:09.831067 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:09.831432 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:09.831924 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:09.832984 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:09.834006 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:09.834955 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:09.835513 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:09.835693 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:09.838629 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:09.838636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:09.838654 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:09.838693 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:09.840991 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:09.841699 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:09.841705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:09.841710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:09.841835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:09.842210 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:09.842213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:09.842214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:09.842624 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:09.842866 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:09.843195 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:59:09.843427 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":169,"threads_started":1}
I20260928 23:59:09.845623 10013 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } }
I20260928 23:59:09.845896 10013 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:09.845978 10013 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9d461c23ea408f9d19261a4313ebed, State: Initialized, Role: FOLLOWER
I20260928 23:59:09.846474 10013 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } }
I20260928 23:59:09.846715 10013 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:09.846778 10013 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:09.846882 10013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:09.847782 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:09.848095 10013 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } }
I20260928 23:59:09.848483 10013 leader_election.cc:302] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc9d461c23ea408f9d19261a4313ebed; no voters:
I20260928 23:59:09.848855 10013 leader_election.cc:288] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:09.848927 10018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:09.849155 10018 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [term 1 LEADER]: Becoming Leader. State: Replica: bc9d461c23ea408f9d19261a4313ebed, State: Running, Role: LEADER
I20260928 23:59:09.849527 10018 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } }
I20260928 23:59:09.850256 10013 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:09.851214 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc9d461c23ea408f9d19261a4313ebed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } } }
I20260928 23:59:09.851382 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:09.851606 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9d461c23ea408f9d19261a4313ebed. Latest consensus state: current_term: 1 leader_uuid: "bc9d461c23ea408f9d19261a4313ebed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9d461c23ea408f9d19261a4313ebed" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 40953 } } }
I20260928 23:59:09.851707 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:09.853083 10032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:09.857036 10032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:09.859777 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:09.850068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42507 (local address 127.20.88.62:40953)
0928 23:59:09.850197 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:09.850201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:09.850420 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:59:09.850563 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:09.850566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:09.850621 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:09.850705 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:09.851687 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:09.852447 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:09.853657 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:09.853895 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:09.853984 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:09.855777 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:09.855782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:09.855785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:09.855814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:09.857674 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:09.858267 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:09.858272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:09.858274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:09.858377 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:09.858770 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:09.858773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:09.858775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:09.859010 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:09.859171 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:09.859487 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 23:59:09.859610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:59:09.863538 10032 catalog_manager.cc:1347] Generated new cluster ID: 9628ec3867e0430f81054ef55edfead3
I20260928 23:59:09.863624 10032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:09.869259 10032 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:09.870205 10032 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:09.878468 10032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc9d461c23ea408f9d19261a4313ebed: Generated new TSK 0
I20260928 23:59:09.879302 10032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:09.884389 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 9945
W20260928 23:59:09.890305 9930 connection.cc:535] client connection to 127.20.88.62:40953 recv error: Network error: failed to read from TLS socket (remote: 127.20.88.62:40953): Connection reset by peer (error 104)
2026-09-28T23:59:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[10044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[10044](info): set up 2 sockets
Sep 28 23:59:09 dist-test-slave-wvpw krb5kdc[10044](info): commencing operation
krb5kdc: starting...
W20260928 23:59:10.945580 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:11 dist-test-slave-wvpw krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:11.088588 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44287
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:11.206238 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:11.206602 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:11.206689 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:11.210930 10057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:11.211038 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:11.211081 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:11.211141 10057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:11.211181 10057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:11.216810 10057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:44287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:11.218019 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:11.219110 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:11.228710 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:11.228714 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:11.228749 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:11.228991 10057 server_base.cc:1034] running on GCE node
I20260928 23:59:11.229746 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:11.230986 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:11.232178 10057 hybrid_clock.cc:648] HybridClock initialized: now 1790639951232154 us; error 39 us; skew 500 ppm
Sep 28 23:59:11 dist-test-slave-wvpw krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639951, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:11.235452 10057 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:11.236552 10057 webserver.cc:460] Webserver started at http://127.20.88.62:34767/ using document root <none> and password file <none>
I20260928 23:59:11.237151 10057 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:11.237214 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:11.237455 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:11.239722 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "605d224713eb4279b7a333a9421e20cf"
format_stamp: "Formatted at 2026-09-28 23:59:11 on dist-test-slave-wvpw"
server_key: "96f56491ae1bb4939806050c2d7a089c"
server_key_iv: "ec061b31cc1435514c72e4ea3b911057"
server_key_version: "encryptionkey@0"
I20260928 23:59:11.240393 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "605d224713eb4279b7a333a9421e20cf"
format_stamp: "Formatted at 2026-09-28 23:59:11 on dist-test-slave-wvpw"
server_key: "96f56491ae1bb4939806050c2d7a089c"
server_key_iv: "ec061b31cc1435514c72e4ea3b911057"
server_key_version: "encryptionkey@0"
I20260928 23:59:11.244647 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:59:11.247524 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:11.249166 10057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:11.249346 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "605d224713eb4279b7a333a9421e20cf"
format_stamp: "Formatted at 2026-09-28 23:59:11 on dist-test-slave-wvpw"
server_key: "96f56491ae1bb4939806050c2d7a089c"
server_key_iv: "ec061b31cc1435514c72e4ea3b911057"
server_key_version: "encryptionkey@0"
I20260928 23:59:11.249482 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:11.275475 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:11.279340 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:11.279585 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:11.287475 10057 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:44287
I20260928 23:59:11.287580 10124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:44287 every 8 connection(s)
I20260928 23:59:11.288640 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:11.291888 10125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:11.296653 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10057
I20260928 23:59:11.296895 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:11.297189 20832 external_mini_cluster.cc:1428] Setting key bcdf4ebb84319eb9b22c2f26075022b6
I20260928 23:59:11.299232 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf: Bootstrap starting.
I20260928 23:59:11.302107 10125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:11.303054 10125 log.cc:824] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:11.305523 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf: No bootstrap required, opened a new log
I20260928 23:59:11.307472 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:11.298794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57787 (local address 127.20.88.62:44287)
0928 23:59:11.299388 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:11.299411 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:11.299464 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:59:11.300223 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:11.300310 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:11.300712 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:11.301208 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:11.301692 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:11.302860 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:11.303915 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:11.304517 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:11.304715 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:11.305706 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:11.305711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:11.305726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:11.306147 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:11.306202 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:11.306226 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:11.306371 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:11.306548 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 23:59:11.306707 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":156,"threads_started":1}
I20260928 23:59:11.310281 10125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } }
I20260928 23:59:11.310575 10125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:11.310683 10125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605d224713eb4279b7a333a9421e20cf, State: Initialized, Role: FOLLOWER
I20260928 23:59:11.310951 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:11.311177 10125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } }
I20260928 23:59:11.311318 10125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:11.311435 10125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:11.311568 10125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:11.312790 10125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } }
I20260928 23:59:11.313184 10125 leader_election.cc:302] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605d224713eb4279b7a333a9421e20cf; no voters:
I20260928 23:59:11.313592 10125 leader_election.cc:288] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:11.313680 10134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:11.313942 10134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [term 1 LEADER]: Becoming Leader. State: Replica: 605d224713eb4279b7a333a9421e20cf, State: Running, Role: LEADER
I20260928 23:59:11.314294 10134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } }
I20260928 23:59:11.315181 10125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:11.316278 10137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605d224713eb4279b7a333a9421e20cf. Latest consensus state: current_term: 1 leader_uuid: "605d224713eb4279b7a333a9421e20cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } } }
I20260928 23:59:11.316310 10136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605d224713eb4279b7a333a9421e20cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605d224713eb4279b7a333a9421e20cf" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 44287 } } }
I20260928 23:59:11.316521 10137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:11.316527 10136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:11.316984 10144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:11.320525 10144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:11.323148 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:11.313667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37099 (local address 127.20.88.62:44287)
0928 23:59:11.313809 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:11.313816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:11.318043 (+ 4227us) server_negotiation.cc:407] Connection header received
0928 23:59:11.318210 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:11.318221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:11.318310 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:11.318412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:11.318888 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:11.319678 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:11.320859 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:11.321206 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:11.321289 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:11.322562 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:11.322566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:11.322569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:11.322584 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:11.322615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:11.322634 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:11.322736 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:11.322917 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 23:59:11.322959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:59:11.327833 10144 catalog_manager.cc:1347] Generated new cluster ID: c60300374df6404f9ff596bf43992d9c
I20260928 23:59:11.327939 10144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:11.362833 10144 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:11.363924 10144 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:11.373318 10144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605d224713eb4279b7a333a9421e20cf: Generated new TSK 0
I20260928 23:59:11.374218 10144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:11.383868 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10057
W20260928 23:59:11.389695 10042 connection.cc:535] client connection to 127.20.88.62:44287 recv error: Network error: failed to read from TLS socket (remote: 127.20.88.62:44287): Connection reset by peer (error 104)
2026-09-28T23:59:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:11 dist-test-slave-wvpw krb5kdc[10156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:11 dist-test-slave-wvpw krb5kdc[10156](info): set up 2 sockets
Sep 28 23:59:11 dist-test-slave-wvpw krb5kdc[10156](info): commencing operation
krb5kdc: starting...
W20260928 23:59:12.445080 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:12 dist-test-slave-wvpw krb5kdc[10156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:12.584681 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:39671
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:33693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:12.716950 10169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:12.717406 10169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:12.717615 10169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:12.723815 10169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:12.723934 10169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:12.723958 10169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:12.723977 10169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:12.723994 10169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:12.730650 10169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:33693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:39671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:12.732038 10169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:12.733333 10169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:12.743386 10175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:12.743387 10174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:12.743387 10177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:12.743719 10169 server_base.cc:1034] running on GCE node
I20260928 23:59:12.744568 10169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:12.745883 10169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:12.747125 10169 hybrid_clock.cc:648] HybridClock initialized: now 1790639952747095 us; error 61 us; skew 500 ppm
Sep 28 23:59:12 dist-test-slave-wvpw krb5kdc[10156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639952, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:12.756992 10169 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:12.758172 10169 webserver.cc:460] Webserver started at http://127.20.88.62:35609/ using document root <none> and password file <none>
I20260928 23:59:12.758841 10169 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:12.758949 10169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:12.759217 10169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:12.761772 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "a7c8e1e0902844e5a72473fe3e07ace5"
format_stamp: "Formatted at 2026-09-28 23:59:12 on dist-test-slave-wvpw"
server_key: "f012a87f74c70a16b09e6078e5d6ef9f"
server_key_iv: "0e332ef9ccea9bd6fe536a851019c0b2"
server_key_version: "encryptionkey@0"
I20260928 23:59:12.762472 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "a7c8e1e0902844e5a72473fe3e07ace5"
format_stamp: "Formatted at 2026-09-28 23:59:12 on dist-test-slave-wvpw"
server_key: "f012a87f74c70a16b09e6078e5d6ef9f"
server_key_iv: "0e332ef9ccea9bd6fe536a851019c0b2"
server_key_version: "encryptionkey@0"
I20260928 23:59:12.766912 10169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:59:12.769928 10184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:12.771570 10169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:59:12.771818 10169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "a7c8e1e0902844e5a72473fe3e07ace5"
format_stamp: "Formatted at 2026-09-28 23:59:12 on dist-test-slave-wvpw"
server_key: "f012a87f74c70a16b09e6078e5d6ef9f"
server_key_iv: "0e332ef9ccea9bd6fe536a851019c0b2"
server_key_version: "encryptionkey@0"
I20260928 23:59:12.771984 10169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:12.777925 10169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:12.781791 10169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:12.782028 10169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:12.790416 10169 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:39671
I20260928 23:59:12.790432 10236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:39671 every 8 connection(s)
I20260928 23:59:12.791698 10169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:12.793931 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10169
I20260928 23:59:12.794092 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:12.794409 20832 external_mini_cluster.cc:1428] Setting key da3882555eed203c9ab44a52cffcc5b5
I20260928 23:59:12.795248 10237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:12.801457 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:12.795988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57615 (local address 127.20.88.62:39671)
0928 23:59:12.796508 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:12.796528 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:12.796574 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:59:12.797595 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:12.797625 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:12.798033 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:12.798475 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:12.799697 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:12.799703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:12.799721 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:12.799736 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:12.799816 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:12.799844 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:12.800021 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:12.800215 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 23:59:12.800426 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":210,"threads_started":1}
I20260928 23:59:12.803346 10237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5: Bootstrap starting.
I20260928 23:59:12.805145 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:12.806411 10237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:12.807471 10237 log.cc:824] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:12.809801 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:12.807751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35977 (local address 127.20.88.62:39671)
0928 23:59:12.807883 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:12.807888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:12.808171 (+ 283us) server_negotiation.cc:407] Connection header received
0928 23:59:12.808380 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:12.808383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:12.808419 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:12.808497 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:12.809372 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:12.809375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:12.809378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:12.809391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:12.809421 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:12.809434 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:12.809529 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:12.809628 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:59:12.809672 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:59:12.810508 10237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5: No bootstrap required, opened a new log
I20260928 23:59:12.816069 10237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } }
I20260928 23:59:12.816489 10237 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:12.816540 10237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c8e1e0902844e5a72473fe3e07ace5, State: Initialized, Role: FOLLOWER
I20260928 23:59:12.817184 10237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } }
I20260928 23:59:12.817374 10237 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:12.817483 10237 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:12.817623 10237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:12.818797 10237 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } }
I20260928 23:59:12.819170 10237 leader_election.cc:302] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7c8e1e0902844e5a72473fe3e07ace5; no voters:
I20260928 23:59:12.819559 10237 leader_election.cc:288] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:12.819669 10247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:12.819875 10247 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [term 1 LEADER]: Becoming Leader. State: Replica: a7c8e1e0902844e5a72473fe3e07ace5, State: Running, Role: LEADER
I20260928 23:59:12.820339 10247 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } }
I20260928 23:59:12.821200 10237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:12.822496 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } } }
I20260928 23:59:12.822657 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:12.822947 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c8e1e0902844e5a72473fe3e07ace5. Latest consensus state: current_term: 1 leader_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8e1e0902844e5a72473fe3e07ace5" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 39671 } } }
I20260928 23:59:12.823091 10256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:12.823117 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:12.826844 10256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:12.833962 10256 catalog_manager.cc:1347] Generated new cluster ID: b8e7f6e881044669b023476a7baf8056
I20260928 23:59:12.834077 10256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:12.843725 10256 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:12.844172 10256 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:12.853412 10256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c8e1e0902844e5a72473fe3e07ace5: Generated new TSK 0
I20260928 23:59:12.854385 10256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:12.864467 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10169
2026-09-28T23:59:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:12 dist-test-slave-wvpw krb5kdc[10268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:12 dist-test-slave-wvpw krb5kdc[10268](info): set up 2 sockets
Sep 28 23:59:12 dist-test-slave-wvpw krb5kdc[10268](info): commencing operation
krb5kdc: starting...
W20260928 23:59:13.933172 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:14.072479 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36377
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:36377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:14.192395 10281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:14.192767 10281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:14.192857 10281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:14.197155 10281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:14.197275 10281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:14.197316 10281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:14.197356 10281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:14.197392 10281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:14.202979 10281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:36377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:14.204177 10281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:14.205257 10281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:59:14.214349 10281 server_base.cc:1034] running on GCE node
W20260928 23:59:14.214253 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:14.214241 10289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:14.214243 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:14.215310 10281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:14.216580 10281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:14.217825 10281 hybrid_clock.cc:648] HybridClock initialized: now 1790639954217760 us; error 82 us; skew 500 ppm
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639954, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:14.221246 10281 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:14.222429 10281 webserver.cc:460] Webserver started at http://127.20.88.62:38901/ using document root <none> and password file <none>
I20260928 23:59:14.223042 10281 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:14.223155 10281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:14.223417 10281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:14.225739 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "43dbcec961ce4fe6806cd8d872523d6c"
format_stamp: "Formatted at 2026-09-28 23:59:14 on dist-test-slave-wvpw"
server_key: "49f3ea41f3b10f9f9c17e5e17c5cf786"
server_key_iv: "d8384ac19cb59ef56787914dcfcb6c32"
server_key_version: "encryptionkey@0"
I20260928 23:59:14.226387 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "43dbcec961ce4fe6806cd8d872523d6c"
format_stamp: "Formatted at 2026-09-28 23:59:14 on dist-test-slave-wvpw"
server_key: "49f3ea41f3b10f9f9c17e5e17c5cf786"
server_key_iv: "d8384ac19cb59ef56787914dcfcb6c32"
server_key_version: "encryptionkey@0"
I20260928 23:59:14.230849 10281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:59:14.233893 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:14.235505 10281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:59:14.235736 10281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "43dbcec961ce4fe6806cd8d872523d6c"
format_stamp: "Formatted at 2026-09-28 23:59:14 on dist-test-slave-wvpw"
server_key: "49f3ea41f3b10f9f9c17e5e17c5cf786"
server_key_iv: "d8384ac19cb59ef56787914dcfcb6c32"
server_key_version: "encryptionkey@0"
I20260928 23:59:14.235886 10281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:14.254539 10281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:14.258505 10281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:14.258782 10281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:14.266284 10281 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:36377
I20260928 23:59:14.266376 10348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:36377 every 8 connection(s)
I20260928 23:59:14.267458 10281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:14.269811 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10281
I20260928 23:59:14.269968 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:14.270305 20832 external_mini_cluster.cc:1428] Setting key 63d9c06bd99b25b5b63dcfcb5676ddac
I20260928 23:59:14.270506 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:14.278118 10349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c: Bootstrap starting.
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:14.280624 10349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:14.281529 10349 log.cc:824] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:14.283782 10349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c: No bootstrap required, opened a new log
I20260928 23:59:14.286365 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:14.271897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58699 (local address 127.20.88.62:36377)
0928 23:59:14.272389 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:14.272411 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:14.272455 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:59:14.273370 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:14.273396 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:14.273858 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:14.274251 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:14.274981 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:14.276506 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:14.277513 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:14.278220 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:14.278382 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:14.281300 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:14.281310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:14.281330 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:14.281371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:14.283238 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:14.283764 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:14.283770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:14.283775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:14.283941 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:14.284269 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:14.284273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:14.284278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:14.284770 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:14.284991 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:14.285280 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 23:59:14.285604 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":144,"threads_started":1}
I20260928 23:59:14.287875 10349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } }
I20260928 23:59:14.288174 10349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:14.288296 10349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43dbcec961ce4fe6806cd8d872523d6c, State: Initialized, Role: FOLLOWER
I20260928 23:59:14.288787 10349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } }
I20260928 23:59:14.288962 10349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:14.289038 10349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:14.289156 10349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:14.289511 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:14.290656 10349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } }
I20260928 23:59:14.291127 10349 leader_election.cc:302] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43dbcec961ce4fe6806cd8d872523d6c; no voters:
I20260928 23:59:14.291523 10349 leader_election.cc:288] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:14.291642 10358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:14.291898 10358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [term 1 LEADER]: Becoming Leader. State: Replica: 43dbcec961ce4fe6806cd8d872523d6c, State: Running, Role: LEADER
I20260928 23:59:14.292343 10358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } }
I20260928 23:59:14.293090 10349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:14.294353 10360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43dbcec961ce4fe6806cd8d872523d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } } }
I20260928 23:59:14.294555 10360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:14.295017 10368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:14.294950 10359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43dbcec961ce4fe6806cd8d872523d6c. Latest consensus state: current_term: 1 leader_uuid: "43dbcec961ce4fe6806cd8d872523d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43dbcec961ce4fe6806cd8d872523d6c" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 36377 } } }
I20260928 23:59:14.295079 10359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:14.298941 10368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:14.302580 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:14.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52415 (local address 127.20.88.62:36377)
0928 23:59:14.292446 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:14.292452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:14.292523 (+ 71us) server_negotiation.cc:407] Connection header received
0928 23:59:14.292737 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:14.292740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:14.292790 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:14.292840 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:14.293877 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:14.294796 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:14.296126 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:14.296519 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:14.296640 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:14.298660 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:14.298664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:14.298667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:14.298697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:14.300589 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:14.301108 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:14.301112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:14.301114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:14.301207 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:14.301532 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:14.301536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:14.301537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:14.301814 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:14.301970 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:14.302260 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 23:59:14.302400 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:59:14.305783 10368 catalog_manager.cc:1347] Generated new cluster ID: e2c0e787a98d4c68b3827cf06135fe27
I20260928 23:59:14.305869 10368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:14.311224 10368 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:14.312192 10368 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:14.323071 10368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43dbcec961ce4fe6806cd8d872523d6c: Generated new TSK 0
I20260928 23:59:14.323923 10368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:14.333711 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10281
2026-09-28T23:59:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10380](info): set up 2 sockets
Sep 28 23:59:14 dist-test-slave-wvpw krb5kdc[10380](info): commencing operation
krb5kdc: starting...
W20260928 23:59:15.421391 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:15 dist-test-slave-wvpw krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:15.556651 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46537
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:36363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:46537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:15.671855 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:15.672343 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:15.672462 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:15.676739 10393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:15.676885 10393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:15.676923 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:15.676963 10393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:15.677009 10393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:15.682801 10393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:36363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:46537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:46537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:15.684028 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:15.685154 10393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:15.694316 10398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:15.694320 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:15.694317 10399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:15.694893 10393 server_base.cc:1034] running on GCE node
I20260928 23:59:15.695605 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:15.696954 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:15.698141 10393 hybrid_clock.cc:648] HybridClock initialized: now 1790639955698116 us; error 33 us; skew 500 ppm
Sep 28 23:59:15 dist-test-slave-wvpw krb5kdc[10380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639955, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:15.701287 10393 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:15.702296 10393 webserver.cc:460] Webserver started at http://127.20.88.62:33323/ using document root <none> and password file <none>
I20260928 23:59:15.702924 10393 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:15.703023 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:15.703323 10393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:15.705598 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "303913acd52442a98fb367643945da2d"
format_stamp: "Formatted at 2026-09-28 23:59:15 on dist-test-slave-wvpw"
server_key: "b682c088554fb02bda34bc0b9600f572"
server_key_iv: "0cdc393759b26d8b87465747266515db"
server_key_version: "encryptionkey@0"
I20260928 23:59:15.706203 10393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "303913acd52442a98fb367643945da2d"
format_stamp: "Formatted at 2026-09-28 23:59:15 on dist-test-slave-wvpw"
server_key: "b682c088554fb02bda34bc0b9600f572"
server_key_iv: "0cdc393759b26d8b87465747266515db"
server_key_version: "encryptionkey@0"
I20260928 23:59:15.710286 10393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:59:15.713171 10408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:15.714643 10393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:59:15.714809 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "303913acd52442a98fb367643945da2d"
format_stamp: "Formatted at 2026-09-28 23:59:15 on dist-test-slave-wvpw"
server_key: "b682c088554fb02bda34bc0b9600f572"
server_key_iv: "0cdc393759b26d8b87465747266515db"
server_key_version: "encryptionkey@0"
I20260928 23:59:15.714942 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:15.732791 10393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:15.737041 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:15.737280 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:15.745514 10393 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:46537
I20260928 23:59:15.745543 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:46537 every 8 connection(s)
I20260928 23:59:15.746541 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:15.749527 10461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:15.754403 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10393
I20260928 23:59:15.754557 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:15.754865 20832 external_mini_cluster.cc:1428] Setting key 9ca8eaa27f659a01f01e9621bc2adf58
I20260928 23:59:15.755689 10461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: Bootstrap starting.
I20260928 23:59:15.758240 10461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:15.759088 10461 log.cc:824] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:15.761467 10461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: No bootstrap required, opened a new log
I20260928 23:59:15.764842 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:15.756390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40751 (local address 127.20.88.62:46537)
0928 23:59:15.756873 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:15.756889 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:15.756928 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:59:15.757596 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:15.757613 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:15.757984 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:15.758285 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:15.758704 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:15.759975 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:15.761253 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:15.761799 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:15.761924 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:15.762898 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:15.762903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:15.762918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:15.763331 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:15.763387 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:15.763412 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:15.763499 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:15.763728 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 23:59:15.763998 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":159,"threads_started":1}
I20260928 23:59:15.765749 10461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } }
I20260928 23:59:15.765972 10461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:15.766031 10461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303913acd52442a98fb367643945da2d, State: Initialized, Role: FOLLOWER
I20260928 23:59:15.766530 10461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } }
I20260928 23:59:15.766692 10461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:15.766783 10461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:15.766909 10461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:15.768049 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:15.768066 10461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } }
I20260928 23:59:15.768450 10461 leader_election.cc:302] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 303913acd52442a98fb367643945da2d; no voters:
I20260928 23:59:15.768899 10461 leader_election.cc:288] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:15.769019 10466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:15.769268 10466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [term 1 LEADER]: Becoming Leader. State: Replica: 303913acd52442a98fb367643945da2d, State: Running, Role: LEADER
I20260928 23:59:15.769683 10466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } }
I20260928 23:59:15.770326 10461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:15.771068 10468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 303913acd52442a98fb367643945da2d. Latest consensus state: current_term: 1 leader_uuid: "303913acd52442a98fb367643945da2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } } }
I20260928 23:59:15.771206 10468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:15.771540 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "303913acd52442a98fb367643945da2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303913acd52442a98fb367643945da2d" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 46537 } } }
I20260928 23:59:15.771632 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d [sys.catalog]: This master's current role is: LEADER
W20260928 23:59:15.775611 10486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:15.775736 10486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:15.775828 10479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:15.775837 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:15.770674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55931 (local address 127.20.88.62:46537)
0928 23:59:15.770789 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:15.770794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:15.771103 (+ 309us) server_negotiation.cc:407] Connection header received
0928 23:59:15.771323 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:15.771330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:15.771390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:15.771448 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:15.771903 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:15.772673 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:15.773978 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:15.774312 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:15.774420 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:15.775331 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:15.775335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:15.775338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:15.775351 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:15.775380 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:15.775393 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:15.775471 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:15.775597 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:59:15.775659 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:59:15.776602 10479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:15.781991 10479 catalog_manager.cc:1347] Generated new cluster ID: 4f2ce678231e43138b54afb0b429a5c4
I20260928 23:59:15.782075 10479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:15.788610 10479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:15.789773 10479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:15.800683 10479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 303913acd52442a98fb367643945da2d: Generated new TSK 0
I20260928 23:59:15.801476 10479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:15.808163 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10393
2026-09-28T23:59:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:15 dist-test-slave-wvpw krb5kdc[10492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:15 dist-test-slave-wvpw krb5kdc[10492](info): set up 2 sockets
Sep 28 23:59:15 dist-test-slave-wvpw krb5kdc[10492](info): commencing operation
krb5kdc: starting...
W20260928 23:59:16.893539 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:16 dist-test-slave-wvpw krb5kdc[10492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:17.028049 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43953
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:43953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:17.143589 10505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:17.144002 10505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:17.144105 10505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:17.148752 10505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:17.148883 10505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:17.148923 10505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:17.148962 10505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:17.148996 10505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:17.154992 10505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:43953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:17.156212 10505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:17.157330 10505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:17.166538 10510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:17.166646 10505 server_base.cc:1034] running on GCE node
W20260928 23:59:17.166584 10511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:17.166584 10513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:17.167454 10505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:17.168720 10505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:17.169912 10505 hybrid_clock.cc:648] HybridClock initialized: now 1790639957169899 us; error 29 us; skew 500 ppm
Sep 28 23:59:17 dist-test-slave-wvpw krb5kdc[10492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639957, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:17.173009 10505 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:17.174060 10505 webserver.cc:460] Webserver started at http://127.20.88.62:42623/ using document root <none> and password file <none>
I20260928 23:59:17.174640 10505 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:17.174705 10505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:17.174940 10505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:17.177382 10505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "ea6ff32a38e34a0691001d7bd8a40716"
format_stamp: "Formatted at 2026-09-28 23:59:17 on dist-test-slave-wvpw"
server_key: "9ef16f5a0a1ca39a0cc2046907a2df95"
server_key_iv: "855a4fa94abd5f41c20b095f5970c18c"
server_key_version: "encryptionkey@0"
I20260928 23:59:17.178115 10505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "ea6ff32a38e34a0691001d7bd8a40716"
format_stamp: "Formatted at 2026-09-28 23:59:17 on dist-test-slave-wvpw"
server_key: "9ef16f5a0a1ca39a0cc2046907a2df95"
server_key_iv: "855a4fa94abd5f41c20b095f5970c18c"
server_key_version: "encryptionkey@0"
I20260928 23:59:17.182456 10505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:59:17.185338 10520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:17.186893 10505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:17.187016 10505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "ea6ff32a38e34a0691001d7bd8a40716"
format_stamp: "Formatted at 2026-09-28 23:59:17 on dist-test-slave-wvpw"
server_key: "9ef16f5a0a1ca39a0cc2046907a2df95"
server_key_iv: "855a4fa94abd5f41c20b095f5970c18c"
server_key_version: "encryptionkey@0"
I20260928 23:59:17.187122 10505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:17.192065 10505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:17.195746 10505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:17.195924 10505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:17.203235 10505 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:43953
I20260928 23:59:17.203307 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:43953 every 8 connection(s)
I20260928 23:59:17.204393 10505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:17.205433 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10505
I20260928 23:59:17.205581 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:17.205900 20832 external_mini_cluster.cc:1428] Setting key b4db4570203689b026e82e432d88f5bf
I20260928 23:59:17.207971 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:17.211612 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:17.207446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46333 (local address 127.20.88.62:43953)
0928 23:59:17.207952 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:17.207967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:17.208005 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:59:17.208815 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:17.208833 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:17.209151 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:17.209523 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:17.210359 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:17.210363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:17.210376 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:17.210386 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:17.210440 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:17.210459 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:17.210644 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:17.210719 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:59:17.210879 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":161,"threads_started":1}
I20260928 23:59:17.214425 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716: Bootstrap starting.
I20260928 23:59:17.214851 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:17.217077 10573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:17.218129 10573 log.cc:824] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:17.218950 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:17.217119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46889 (local address 127.20.88.62:43953)
0928 23:59:17.217263 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:17.217267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:17.217455 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:59:17.217556 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:17.217559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:17.217608 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:17.217692 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:17.218484 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:17.218491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:17.218497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:59:17.218513 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:17.218546 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:59:17.218570 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:17.218662 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:17.218770 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:59:17.218803 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:59:17.220589 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716: No bootstrap required, opened a new log
I20260928 23:59:17.224344 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } }
I20260928 23:59:17.224586 10573 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:59:17.224629 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6ff32a38e34a0691001d7bd8a40716, State: Initialized, Role: FOLLOWER
I20260928 23:59:17.225009 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } }
I20260928 23:59:17.225127 10573 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:59:17.225173 10573 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:59:17.225244 10573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:17.226197 10573 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } }
I20260928 23:59:17.226496 10573 leader_election.cc:302] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea6ff32a38e34a0691001d7bd8a40716; no voters:
I20260928 23:59:17.226760 10573 leader_election.cc:288] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:59:17.226897 10583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:17.227139 10583 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [term 1 LEADER]: Becoming Leader. State: Replica: ea6ff32a38e34a0691001d7bd8a40716, State: Running, Role: LEADER
I20260928 23:59:17.227523 10583 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } }
I20260928 23:59:17.228120 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:17.229343 10584 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea6ff32a38e34a0691001d7bd8a40716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } } }
I20260928 23:59:17.229480 10584 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:17.229880 10592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:17.229863 10585 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea6ff32a38e34a0691001d7bd8a40716. Latest consensus state: current_term: 1 leader_uuid: "ea6ff32a38e34a0691001d7bd8a40716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6ff32a38e34a0691001d7bd8a40716" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 43953 } } }
I20260928 23:59:17.230017 10585 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:17.233297 10592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:17.239420 10592 catalog_manager.cc:1347] Generated new cluster ID: 1b1927b69f9c4dc789308ccd3f2bbf29
I20260928 23:59:17.239504 10592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:17.255596 10592 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:17.256052 10592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:17.265724 10592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea6ff32a38e34a0691001d7bd8a40716: Generated new TSK 0
I20260928 23:59:17.266577 10592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:59:17.273869 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10505
2026-09-28T23:59:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1465 ms)
[----------] 18 tests from AuthTokenIssuingTest (26327 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:17 dist-test-slave-wvpw krb5kdc[10604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:17 dist-test-slave-wvpw krb5kdc[10604](info): set up 2 sockets
Sep 28 23:59:17 dist-test-slave-wvpw krb5kdc[10604](info): commencing operation
krb5kdc: starting...
W20260928 23:59:18.337190 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:18.477224 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:33139
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:18.598340 10617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:18.598685 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:18.598716 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:18.598764 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:18.602969 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:18.603052 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:18.603072 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:18.603091 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:18.603106 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:18.609491 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:33139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:18.610651 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:18.611696 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:18.620784 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:18.620784 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:18.620818 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:18.621104 10617 server_base.cc:1034] running on GCE node
I20260928 23:59:18.621759 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:18.622984 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:18.624147 10617 hybrid_clock.cc:648] HybridClock initialized: now 1790639958624130 us; error 32 us; skew 500 ppm
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:18.632611 10617 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:18.633683 10617 webserver.cc:460] Webserver started at http://127.20.88.62:36907/ using document root <none> and password file <none>
I20260928 23:59:18.634310 10617 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:18.634379 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:18.634558 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:18.636775 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "1c94dd7af13047d9a19735b09f3963b2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "6a43fb8e795c52e397340841f1672c8b"
server_key_iv: "a26ad973ecf12dcd9f2f6ccdd1b809cd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.637331 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "1c94dd7af13047d9a19735b09f3963b2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "6a43fb8e795c52e397340841f1672c8b"
server_key_iv: "a26ad973ecf12dcd9f2f6ccdd1b809cd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.641274 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:59:18.643821 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:18.645285 10617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:18.645403 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "1c94dd7af13047d9a19735b09f3963b2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "6a43fb8e795c52e397340841f1672c8b"
server_key_iv: "a26ad973ecf12dcd9f2f6ccdd1b809cd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.645509 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:18.659375 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:18.662617 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:18.662797 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:18.670204 10617 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:33139
I20260928 23:59:18.670535 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:33139 every 8 connection(s)
I20260928 23:59:18.671239 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:18.674145 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:18.674909 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10617
I20260928 23:59:18.675061 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:18.675370 20832 external_mini_cluster.cc:1428] Setting key 4069d1a4537678c9bd1e226bdb4d06a1
I20260928 23:59:18.679040 10685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.88.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.61@KRBTEST.COM" created.
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:18.697141 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.683060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.20.88.62:33139)
0928 23:59:18.683416 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:18.683428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:18.683474 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:59:18.683985 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:18.683989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:18.684196 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:18.684434 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:18.685405 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.686365 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.687851 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.688344 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.688543 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.691404 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:18.691411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:18.691424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:18.691447 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:18.693425 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.694015 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.694022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.694025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.694144 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.694517 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.694523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.694525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.694986 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:18.695150 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:18.695574 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 23:59:18.695869 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260928 23:59:18.697141 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.681946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33139 (local address 127.0.0.1:45982)
0928 23:59:18.682559 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.682642 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:18.683666 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:18.684431 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:18.684437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:18.684891 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:18.685249 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.685265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.686556 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.686561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.687680 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.687696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.688534 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.688541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.688685 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.689326 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:18.689343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:18.691218 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:18.693611 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.693620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.693633 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.693888 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.694251 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.694254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.694257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.694410 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.695129 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:18.695136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:18.695291 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 23:59:18.695486 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 23:59:18.695926 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":125,"thread_start_us":193,"threads_started":1}
I20260928 23:59:18.699961 10685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:18.700944 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.700655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43948)
0928 23:59:18.700793 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.700872 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 23:59:18.701272 10634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
I20260928 23:59:18.702375 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.702128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43950)
0928 23:59:18.702249 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.702311 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:59:18.702562 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:34539: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
I20260928 23:59:18.704200 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:34539
--webserver_interface=127.20.88.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:59:18.751893 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } attempt: 1
I20260928 23:59:18.752962 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.752571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43964)
0928 23:59:18.752784 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.752868 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 23:59:18.753839 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.753502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43970)
0928 23:59:18.753707 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.753766 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:59:18.754029 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:34539: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
I20260928 23:59:18.822266 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } attempt: 2
I20260928 23:59:18.823179 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.822836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43974)
0928 23:59:18.823014 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.823087 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 23:59:18.823916 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.823704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43980)
0928 23:59:18.823811 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.823860 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:59:18.824067 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:34539: Network error: Client connection negotiation failed: client connection to 127.20.88.61:34539: connect: Connection refused (error 111)
W20260928 23:59:18.825115 10692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:18.825407 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:18.825477 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:18.825552 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:18.829818 10692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:18.829885 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:18.829905 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:18.829921 10692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:18.829965 10692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:18.835700 10692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.88.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:18.836951 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:18.837993 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:18.846915 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:18.846954 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:18.847070 10692 server_base.cc:1034] running on GCE node
W20260928 23:59:18.846971 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:18.847906 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:18.849179 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:18.850342 10692 hybrid_clock.cc:648] HybridClock initialized: now 1790639958850311 us; error 50 us; skew 500 ppm
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:18.853266 10692 init.cc:377] Logged in from keytab as kudu/127.20.88.61@KRBTEST.COM (short username kudu)
I20260928 23:59:18.854195 10692 webserver.cc:460] Webserver started at http://127.20.88.61:41527/ using document root <none> and password file <none>
I20260928 23:59:18.854796 10692 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:18.854861 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:18.855108 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:18.857339 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data/instance:
uuid: "7044eb5a03af4926b3a271be847a82c2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "720d54937d1cec2446524c34666663bd"
server_key_iv: "0a7dbf753512686106664004390ce7dd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.857944 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal/instance:
uuid: "7044eb5a03af4926b3a271be847a82c2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "720d54937d1cec2446524c34666663bd"
server_key_iv: "0a7dbf753512686106664004390ce7dd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.861941 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:59:18.864580 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:18.865861 10692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:59:18.866012 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "7044eb5a03af4926b3a271be847a82c2"
format_stamp: "Formatted at 2026-09-28 23:59:18 on dist-test-slave-wvpw"
server_key: "720d54937d1cec2446524c34666663bd"
server_key_iv: "0a7dbf753512686106664004390ce7dd"
server_key_version: "encryptionkey@0"
I20260928 23:59:18.866130 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:18.891268 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:18.894974 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:18.895213 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:18.902895 10692 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.61:34539
I20260928 23:59:18.903167 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.61:34539 every 8 connection(s)
I20260928 23:59:18.903992 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
I20260928 23:59:18.907301 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:18.911813 10760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:18.913276 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10692
I20260928 23:59:18.913399 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-1/wal/instance
I20260928 23:59:18.913723 20832 external_mini_cluster.cc:1428] Setting key 58277eb95736c60e6c78661e4c4c4997
I20260928 23:59:18.915232 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } attempt: 3
WARNING: no policy specified for kudu/127.20.88.60@KRBTEST.COM; defaulting to no policy
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.60@KRBTEST.COM" created.
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:18.927726 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.915728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:43986)
0928 23:59:18.915917 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.915932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:18.916026 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:18.917342 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:18.917347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:18.917531 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:18.917676 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.917682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.919463 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.919470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.920365 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.920376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.921289 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.921294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.921429 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.921963 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:18.921984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:18.923521 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:18.925522 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.925524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.925526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.925738 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.926192 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.926198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.926203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.926307 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.927128 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:18.927131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:18.927256 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 23:59:18.927385 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 23:59:18.927540 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"spinlock_wait_cycles":5120}
I20260928 23:59:18.929122 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.915830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.20.88.61:34539)
0928 23:59:18.916304 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:18.916316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:18.916360 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:59:18.916841 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:18.916870 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:18.917141 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:18.917821 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:18.918251 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.919335 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.920499 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.921130 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.921693 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.923657 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:18.923662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:18.923675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:18.923702 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:18.925404 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.925860 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.925887 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.925896 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.926066 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.926427 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.926430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.926431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.927012 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:18.927164 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:18.927555 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 23:59:18.927921 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":343,"thread_start_us":161,"threads_started":1}
I20260928 23:59:18.931432 10685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:18.931989 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.914428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33139 (local address 127.0.0.1:45994)
0928 23:59:18.915017 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.915085 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:18.916808 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:18.917307 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:18.917319 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:18.917829 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:18.918463 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.918483 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.919686 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.919689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.920732 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.920741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.921273 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.921277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.921437 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.922164 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:18.922181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:18.927029 (+ 4848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:18.929814 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.929821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.929826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.930110 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.930467 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.930470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.930472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.930566 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.931149 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:18.931160 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:18.931315 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 23:59:18.931453 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 23:59:18.931666 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":134,"threads_started":1}
I20260928 23:59:18.932221 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.913978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.20.88.62:33139)
0928 23:59:18.914129 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:18.914134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:18.915229 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 23:59:18.917066 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:18.917069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:18.917120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:18.917233 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:18.918657 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.919538 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.920856 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.921104 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.921229 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.927226 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:18.927233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:18.927239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:18.927272 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:18.929703 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.930240 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.930247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.930252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.930345 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.930711 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.930717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.930722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.930993 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:18.931832 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:18.931937 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:59:18.932063 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":1792}
I20260928 23:59:18.932221 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.931959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35710)
0928 23:59:18.932131 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.932177 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 23:59:18.933207 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.932888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35726)
0928 23:59:18.933066 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.933139 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 23:59:18.933360 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
I20260928 23:59:18.933578 10760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:59:18 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:59:18.942700 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.934099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:44002)
0928 23:59:18.934203 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.934213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:18.934299 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:18.934554 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:18.934557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:18.934728 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:18.934814 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.934819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.935540 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.935543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.936067 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:18.936073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.936635 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.936638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:18.936767 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.937218 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:18.937228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:18.938939 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:18.940983 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.940986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.940988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.941308 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.941608 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:18.941610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:18.941612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:18.941749 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:18.942193 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:18.942196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:18.942285 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 23:59:18.942390 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:59:18.942564 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":21}
I20260928 23:59:18.942909 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.934256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.20.88.61:34539)
0928 23:59:18.934380 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:18.934384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:18.934395 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:59:18.934422 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:18.934425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:18.934463 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:18.934545 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:18.934920 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.935430 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.936175 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:18.936527 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:18.936638 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:18.939124 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:18.939127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:18.939129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:18.939152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:18.940846 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.941409 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.941411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.941413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.941486 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:18.941871 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:18.941874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:18.941876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:18.942067 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:18.942145 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:18.942506 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 23:59:18.942651 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":49}
I20260928 23:59:18.943650 10760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:18.944525 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.944195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35730)
0928 23:59:18.944373 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.944465 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 23:59:18.944772 10709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
I20260928 23:59:18.945634 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.945424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35732)
0928 23:59:18.945533 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.945584 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:59:18.945905 10760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
I20260928 23:59:18.947361 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:35865
--webserver_interface=127.20.88.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:59:18.964516 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 1
I20260928 23:59:18.965495 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.965182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35742)
0928 23:59:18.965341 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.965416 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 23:59:18.966444 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.966200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35756)
0928 23:59:18.966325 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.966381 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 23:59:18.966702 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
I20260928 23:59:18.995253 10760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 1
I20260928 23:59:18.996315 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.995906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35770)
0928 23:59:18.996093 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.996171 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 23:59:18.997574 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:18.997322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35780)
0928 23:59:18.997442 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:18.997501 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:59:18.997781 10760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
I20260928 23:59:19.041919 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 2
I20260928 23:59:19.042948 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.042564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35784)
0928 23:59:19.042778 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.042856 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 23:59:19.043851 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.043576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35790)
0928 23:59:19.043711 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.043774 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:59:19.044060 10685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
W20260928 23:59:19.064110 10767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:19.064440 10767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:19.064465 10767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:19.064563 10767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:59:19.065977 10760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 2
I20260928 23:59:19.066912 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.066538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35804)
0928 23:59:19.066743 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.066818 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 23:59:19.067682 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.067460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35820)
0928 23:59:19.067568 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.067618 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:59:19.067835 10760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:35865: Network error: Client connection negotiation failed: client connection to 127.20.88.60:35865: connect: Connection refused (error 111)
W20260928 23:59:19.069416 10767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:19.069476 10767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:19.069495 10767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:19.069536 10767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:19.069569 10767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:19.075443 10767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:33139,127.20.88.61:34539,127.20.88.60:35865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.88.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:19.076715 10767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:19.077819 10767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:19.087487 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:19.087487 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:19.087680 10767 server_base.cc:1034] running on GCE node
W20260928 23:59:19.087487 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:19.088605 10767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:19.089804 10767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:19.090986 10767 hybrid_clock.cc:648] HybridClock initialized: now 1790639959090954 us; error 48 us; skew 500 ppm
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:19.094455 10767 init.cc:377] Logged in from keytab as kudu/127.20.88.60@KRBTEST.COM (short username kudu)
I20260928 23:59:19.095407 10767 webserver.cc:460] Webserver started at http://127.20.88.60:38123/ using document root <none> and password file <none>
I20260928 23:59:19.095983 10767 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:19.096076 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:19.096380 10767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:19.098474 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data/instance:
uuid: "0b11bcbcff654b2c9a3c0226cf2e702e"
format_stamp: "Formatted at 2026-09-28 23:59:19 on dist-test-slave-wvpw"
server_key: "7a902587d62a5ea715c62c7a9f9ee56b"
server_key_iv: "e4d0a1a5bdb0334739deb3aa3fe56796"
server_key_version: "encryptionkey@0"
I20260928 23:59:19.099061 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal/instance:
uuid: "0b11bcbcff654b2c9a3c0226cf2e702e"
format_stamp: "Formatted at 2026-09-28 23:59:19 on dist-test-slave-wvpw"
server_key: "7a902587d62a5ea715c62c7a9f9ee56b"
server_key_iv: "e4d0a1a5bdb0334739deb3aa3fe56796"
server_key_version: "encryptionkey@0"
I20260928 23:59:19.102978 10767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 23:59:19.105719 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:19.107023 10767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:59:19.107195 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "0b11bcbcff654b2c9a3c0226cf2e702e"
format_stamp: "Formatted at 2026-09-28 23:59:19 on dist-test-slave-wvpw"
server_key: "7a902587d62a5ea715c62c7a9f9ee56b"
server_key_iv: "e4d0a1a5bdb0334739deb3aa3fe56796"
server_key_version: "encryptionkey@0"
I20260928 23:59:19.107324 10767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:19.122694 10767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:19.126525 10767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:19.126757 10767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:19.134025 10767 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.60:35865
I20260928 23:59:19.134420 10834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.60:35865 every 8 connection(s)
I20260928 23:59:19.135141 10767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
I20260928 23:59:19.138161 10835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:19.142100 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:19.143286 10685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 3
I20260928 23:59:19.144675 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10767
I20260928 23:59:19.144824 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639669102918-20832-0/minicluster-data/master-2/wal/instance
I20260928 23:59:19.145155 20832 external_mini_cluster.cc:1428] Setting key 50ba0fadfc00748d3fec0650b5b4cf41
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:59:19.157958 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.144062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.20.88.62:33139)
0928 23:59:19.144155 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.144162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.145331 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 23:59:19.146436 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.146442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.146502 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.146559 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.147553 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.148069 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.149563 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.149877 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.149989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.153026 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.153030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.153033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.153064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.155040 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.155866 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.155869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.155871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.155957 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.156441 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.156444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.156446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.156826 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.156927 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.157625 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.157781 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:59:19.158463 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.143903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35836)
0928 23:59:19.144101 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.144118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:19.144300 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:19.146639 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:19.146642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:19.146868 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:19.146981 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.146988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.148814 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.148820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.149655 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.149668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.150549 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.150553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.150633 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.151344 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:19.151362 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:19.153658 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:19.156376 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.156382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.156387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.156625 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.157001 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.157006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.157007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.157091 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.157614 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:19.157618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:19.157727 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:59:19.157818 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:59:19.157948 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 23:59:19.159066 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.145874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.20.88.60:35865)
0928 23:59:19.146187 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.146195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.146374 (+ 179us) server_negotiation.cc:407] Connection header received
0928 23:59:19.146500 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.146502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.146542 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.147712 (+ 1170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.147804 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.148759 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.149967 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.150386 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.150497 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.154416 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.154422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.154425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.154450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.156774 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.157282 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.157286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.157288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.157383 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.157682 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.157685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.157687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.157883 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.158078 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.158464 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.158609 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":244,"spinlock_wait_cycles":896,"thread_start_us":112,"threads_started":1}
I20260928 23:59:19.159075 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.144537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:33139 (local address 127.0.0.1:46008)
0928 23:59:19.145173 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.145223 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:19.146177 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:19.146788 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:19.146796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:19.147227 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:19.147417 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.147427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.148183 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.148189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.149392 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.149401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.150104 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.150107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.150291 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.151004 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:19.151030 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:19.152877 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:19.155190 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.155201 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.155214 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.155715 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.156068 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.156072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.156074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.156296 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.156944 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:19.156953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:19.157324 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 23:59:19.157565 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 23:59:19.157948 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":3,"spinlock_wait_cycles":61312,"thread_start_us":174,"threads_started":1}
I20260928 23:59:19.159044 10760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } attempt: 3
I20260928 23:59:19.159813 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.144537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.20.88.60:35865)
0928 23:59:19.145108 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.145129 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.145174 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:59:19.146158 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.146182 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.146468 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.146719 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.147183 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.148684 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.149776 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.150404 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.150832 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.153843 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.153854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.153873 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.153904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.156272 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.156752 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.156758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.156760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.156853 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.157192 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.157195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.157197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.157471 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.158989 (+ 1518us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.159091 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.159624 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":392,"thread_start_us":153,"threads_started":1}
I20260928 23:59:19.160657 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:19.164045 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: Bootstrap starting.
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:59:19.167112 10685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:19.168162 10685 log.cc:824] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:19.171109 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.161172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:34539 (local address 127.0.0.1:44010)
0928 23:59:19.161257 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.161272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:19.161373 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:19.161700 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:19.161702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:19.161900 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:19.162029 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.162039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.163079 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.163085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.163933 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.163944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.164552 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.164654 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.165196 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:19.165207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:19.166908 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:19.169240 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.169243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.169245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.169673 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.170021 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.170024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.170026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.170122 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.170565 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:19.170568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:19.170676 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:59:19.170771 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:59:19.170903 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:59:19.171640 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.161320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.20.88.61:34539)
0928 23:59:19.161474 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.161478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.161491 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:59:19.161551 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.161554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.161599 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.161676 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.162157 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.162949 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164078 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.164428 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.167072 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.167077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.167081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.167143 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.169122 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.169812 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.169816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.169818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.169912 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.170223 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.170227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.170229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.170433 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.170522 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.171351 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.171473 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:59:19.171849 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.161178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35860)
0928 23:59:19.161375 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.161400 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:19.161491 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:19.161880 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:19.161884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:19.162051 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:19.162161 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.162167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.163161 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.163165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.163972 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.163984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.164664 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.164755 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.165348 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:19.165362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:19.167488 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:19.169918 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.169922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.169924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.170192 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.170609 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.170613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.170615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.170733 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.171253 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:19.171258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:19.171366 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:59:19.171446 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:59:19.171612 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 23:59:19.172317 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: No bootstrap required, opened a new log
I20260928 23:59:19.172680 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.160560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.20.88.60:35865)
0928 23:59:19.160804 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.160808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.161602 (+ 794us) server_negotiation.cc:407] Connection header received
0928 23:59:19.161697 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.161699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.161748 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.162271 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.162345 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.163055 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164104 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.164538 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.164927 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.167633 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.167639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.167643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.167676 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.169766 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.170329 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.170333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.170335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.170430 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.170883 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.170886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.170888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.171114 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.172318 (+ 1204us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.172389 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.172510 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 23:59:19.172680 10835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:19.176446 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: Bootstrap starting.
I20260928 23:59:19.178184 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.178615 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c94dd7af13047d9a19735b09f3963b2, State: Initialized, Role: FOLLOWER
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639959, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:59:19.179231 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.179962 10760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:19.180615 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.180989 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.181135 10760 log.cc:824] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:19.181420 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:19.182554 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.168771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.20.88.62:33139)
0928 23:59:19.168949 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.168953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.168970 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:59:19.169018 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.169023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.169076 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.169174 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.170196 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.170924 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.172194 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.172537 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.172653 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.175751 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.175755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.175758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.175789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.178224 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.178814 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.178819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.178821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.178919 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.179287 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.179290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.179292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.179538 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.179708 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.182210 (+ 2502us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.182371 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 23:59:19.184077 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.173208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:35865 (local address 127.0.0.1:35868)
0928 23:59:19.173517 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:19.173532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:19.173626 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:19.174056 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:19.174059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:19.174241 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:19.174355 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.174362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.175431 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.175434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.176410 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:19.176423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.177276 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.177281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:19.177365 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.177796 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:19.177811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:19.179501 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:19.181921 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.181924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.181927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.182210 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.182572 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:19.182578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:19.182582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:19.182690 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:19.183483 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:19.183489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:19.183606 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 23:59:19.183708 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:59:19.183871 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260928 23:59:19.184919 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: No bootstrap required, opened a new log
I20260928 23:59:19.184952 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.173417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.20.88.60:35865)
0928 23:59:19.173780 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.173784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.173807 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:59:19.173838 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.173841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.173884 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.173958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.174476 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.175305 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.176546 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.177009 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.177136 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.179664 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.179668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.179671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.179700 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.181649 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.182331 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.182335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.182337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.182440 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.183089 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.183093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.183095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.183357 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.184623 (+ 1266us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.184720 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.184824 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260928 23:59:19.188205 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: Bootstrap starting.
I20260928 23:59:19.190176 10760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.190419 10760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7044eb5a03af4926b3a271be847a82c2, State: Initialized, Role: FOLLOWER
I20260928 23:59:19.190841 10760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.191439 10835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: Neither blocks nor log segments found. Creating new log.
Sep 28 23:59:19 dist-test-slave-wvpw krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:59:19.192499 10835 log.cc:824] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:19.192731 10760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:19.188798 10617 master_runner.cc:186] Error getting master registration for 127.20.88.61:34539: OK, instance_id { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" instance_seqno: 1790639958900812 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:59:19.193311 10617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 23:59:19.197475 10854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:19.197571 10854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:19.197894 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.184568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.20.88.61:34539)
0928 23:59:19.184989 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.184994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.185010 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:59:19.185064 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.185067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.185113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.185194 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.186922 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.187833 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.189057 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.189397 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.189512 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.193069 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.193073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.193076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.193107 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.195193 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.195821 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.195826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.195828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.195929 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.196326 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.196330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.196332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.196590 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.196789 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.197521 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.197679 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260928 23:59:19.200117 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: No bootstrap required, opened a new log
I20260928 23:59:19.203758 10835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.203990 10835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b11bcbcff654b2c9a3c0226cf2e702e, State: Initialized, Role: FOLLOWER
I20260928 23:59:19.204411 10835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.197770 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.204908 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.205888 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:19.206050 10835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:19.207456 10658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1c94dd7af13047d9a19735b09f3963b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46022
I20260928 23:59:19.207751 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:59:19.207891 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:19.210027 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
I20260928 23:59:19.211920 10734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c94dd7af13047d9a19735b09f3963b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7044eb5a03af4926b3a271be847a82c2"
I20260928 23:59:19.212455 10809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c94dd7af13047d9a19735b09f3963b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e"
I20260928 23:59:19.201202 10692 master_runner.cc:186] Error getting master registration for 127.20.88.60:35865: OK, instance_id { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" instance_seqno: 1790639959132107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:59:19.212698 10809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:19.212862 10692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:59:19.214298 10809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c94dd7af13047d9a19735b09f3963b2 in term 1.
I20260928 23:59:19.214383 10734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:19.215163 10634 leader_election.cc:302] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b11bcbcff654b2c9a3c0226cf2e702e, 1c94dd7af13047d9a19735b09f3963b2; no voters:
I20260928 23:59:19.215490 10842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:19.215747 10842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c94dd7af13047d9a19735b09f3963b2, State: Running, Role: LEADER
W20260928 23:59:19.215840 10867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:19.215912 10867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:19.215970 10734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c94dd7af13047d9a19735b09f3963b2 in term 1.
I20260928 23:59:19.216133 10842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } }
W20260928 23:59:19.212172 10880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:19.217350 10880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:19.219183 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c94dd7af13047d9a19735b09f3963b2. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.219326 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:19.220436 10869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.220651 10869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.221127 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7044eb5a03af4926b3a271be847a82c2 (127.20.88.61:34539), 0b11bcbcff654b2c9a3c0226cf2e702e (127.20.88.60:35865)
I20260928 23:59:19.223315 10767 master_runner.cc:263] No leader master found from master 0b11bcbcff654b2c9a3c0226cf2e702e
I20260928 23:59:19.223387 10767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:59:19.224639 10885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:19.225749 10885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:19.231796 10809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [term 1 FOLLOWER]: Refusing update from remote peer 1c94dd7af13047d9a19735b09f3963b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:59:19.232597 10883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:59:19.237360 10869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c94dd7af13047d9a19735b09f3963b2. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.237541 10869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.238794 10734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [term 1 FOLLOWER]: Refusing update from remote peer 1c94dd7af13047d9a19735b09f3963b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:59:19.239506 10883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:59:19.244359 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c94dd7af13047d9a19735b09f3963b2. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.244517 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.250819 10883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.250993 10883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:19.252435 10855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.252604 10855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.253249 10890 mvcc.cc:205] Tried to move back new op lower bound from 7334461273005625344 to 7334461272957399040. Current Snapshot: MvccSnapshot[applied={T|T < 7334461273005625344}]
I20260928 23:59:19.256390 10896 mvcc.cc:205] Tried to move back new op lower bound from 7334461273005625344 to 7334461272957399040. Current Snapshot: MvccSnapshot[applied={T|T < 7334461273005625344}]
I20260928 23:59:19.257331 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.257490 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:19.258986 10869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c94dd7af13047d9a19735b09f3963b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c94dd7af13047d9a19735b09f3963b2" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 33139 } } peers { permanent_uuid: "7044eb5a03af4926b3a271be847a82c2" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 34539 } } peers { permanent_uuid: "0b11bcbcff654b2c9a3c0226cf2e702e" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 35865 } } }
I20260928 23:59:19.259142 10869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:19.260031 10894 mvcc.cc:205] Tried to move back new op lower bound from 7334461273005625344 to 7334461272957399040. Current Snapshot: MvccSnapshot[applied={T|T < 7334461273005625344}]
I20260928 23:59:19.265578 10885 catalog_manager.cc:1347] Generated new cluster ID: 113932ba486a4be2b428422cf5dd8195
I20260928 23:59:19.265661 10885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:19.271039 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.231857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.20.88.61:34539)
0928 23:59:19.231983 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.231988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.240592 (+ 8604us) server_negotiation.cc:407] Connection header received
0928 23:59:19.241062 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.241066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.241125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.241182 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.245158 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.246052 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.253242 (+ 7190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.253645 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.253741 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.259672 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.259676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.259680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.259736 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.261741 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.267341 (+ 5600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.267345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.267347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.267445 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.268585 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.268590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.268592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.268856 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.269023 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.270723 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.270867 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:59:19.272991 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.232777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.20.88.60:35865)
0928 23:59:19.233226 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.233231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.243771 (+ 10540us) server_negotiation.cc:407] Connection header received
0928 23:59:19.243952 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.243956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.244011 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.244086 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.249661 (+ 5575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.250528 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.254550 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.254967 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.255063 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.265099 (+ 10036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.265104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.265107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.265139 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.266800 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.267768 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.267771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.267773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.267854 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.269379 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.269383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.269385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.269598 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.269705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.272701 (+ 2996us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.272834 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260928 23:59:19.274816 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:19.231325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.20.88.62:33139)
0928 23:59:19.231457 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.231462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.240965 (+ 9503us) server_negotiation.cc:407] Connection header received
0928 23:59:19.242458 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.242462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.242518 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.242573 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.246208 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.247067 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.251884 (+ 4817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:19.252323 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:19.252411 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:19.262376 (+ 9965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:19.262381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:19.262384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:19.262420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:19.264345 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.268160 (+ 3815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.268164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.268166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.268285 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:19.269995 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:19.269999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:19.270001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:19.270254 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:19.270355 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:19.274522 (+ 4167us) server_negotiation.cc:299] Negotiation successful
0928 23:59:19.274660 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:59:19.295027 10885 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:19.296520 10885 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:19.316082 10885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c94dd7af13047d9a19735b09f3963b2: Generated new TSK 0
I20260928 23:59:19.317126 10885 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:59:19.352130 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.348549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.20.88.61:34539)
0928 23:59:19.348747 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.348752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.350897 (+ 2145us) server_negotiation.cc:407] Connection header received
0928 23:59:19.351125 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.351129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.351187 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.351248 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.351975 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44046: BlockingRecv error: recv got EOF from 127.0.0.1:44046 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260928 23:59:19.354008 10839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.349423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.20.88.60:35865)
0928 23:59:19.349551 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.349555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.352734 (+ 3179us) server_negotiation.cc:407] Connection header received
0928 23:59:19.353163 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.353167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.353226 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.353285 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.353867 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35880: BlockingRecv error: recv got EOF from 127.0.0.1:35880 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 23:59:19.355257 10689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.350217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.20.88.62:33139)
0928 23:59:19.350343 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.350348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.353066 (+ 2718us) server_negotiation.cc:407] Connection header received
0928 23:59:19.354418 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.354422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.354483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.354539 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.355126 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46050: BlockingRecv error: recv got EOF from 127.0.0.1:46050 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:59:19.360973 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.359117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.20.88.61:34539)
0928 23:59:19.359252 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.359257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.359760 (+ 503us) server_negotiation.cc:407] Connection header received
0928 23:59:19.360021 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.360025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.360114 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.360174 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.360846 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44058: BlockingRecv error: recv got EOF from 127.0.0.1:44058 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 23:59:19.367041 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.365307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.20.88.61:34539)
0928 23:59:19.365433 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.365438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.366003 (+ 565us) server_negotiation.cc:407] Connection header received
0928 23:59:19.366213 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.366217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.366274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.366330 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.366913 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44064: BlockingRecv error: recv got EOF from 127.0.0.1:44064 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:59:19.378288 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.376385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.20.88.61:34539)
0928 23:59:19.376519 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.376524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.377191 (+ 667us) server_negotiation.cc:407] Connection header received
0928 23:59:19.377425 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.377429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.377485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.377538 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.378162 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44078: BlockingRecv error: recv got EOF from 127.0.0.1:44078 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:59:19.389748 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.385295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.20.88.61:34539)
0928 23:59:19.385462 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.385467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.388613 (+ 3146us) server_negotiation.cc:407] Connection header received
0928 23:59:19.388865 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.388869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.388934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.388991 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.389598 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44082: BlockingRecv error: recv got EOF from 127.0.0.1:44082 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 23:59:19.402452 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.400626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.20.88.61:34539)
0928 23:59:19.400775 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.400779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.401387 (+ 608us) server_negotiation.cc:407] Connection header received
0928 23:59:19.401614 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.401617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.401675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.401745 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.402329 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44098: BlockingRecv error: recv got EOF from 127.0.0.1:44098 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:59:19.423426 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.421558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.20.88.61:34539)
0928 23:59:19.421708 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.421713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.422328 (+ 615us) server_negotiation.cc:407] Connection header received
0928 23:59:19.422551 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.422555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.422612 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.422682 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.423293 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44112: BlockingRecv error: recv got EOF from 127.0.0.1:44112 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:59:19.460325 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.458708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.20.88.61:34539)
0928 23:59:19.458855 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.458862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.459200 (+ 338us) server_negotiation.cc:407] Connection header received
0928 23:59:19.459371 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.459374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.459427 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.459502 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.460104 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44120: BlockingRecv error: recv got EOF from 127.0.0.1:44120 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 23:59:19.527555 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.526503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.20.88.61:34539)
0928 23:59:19.526642 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.526646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.526752 (+ 106us) server_negotiation.cc:407] Connection header received
0928 23:59:19.526882 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.526885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.526935 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.527018 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.527452 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44134: BlockingRecv error: recv got EOF from 127.0.0.1:44134 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:59:19.659338 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.658056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.20.88.61:34539)
0928 23:59:19.658209 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.658213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.658296 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:59:19.658481 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.658484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.658540 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.658626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.659167 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44150: BlockingRecv error: recv got EOF from 127.0.0.1:44150 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 23:59:19.918942 10764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:19.917737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.20.88.61:34539)
0928 23:59:19.917899 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:19.917904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:19.918020 (+ 116us) server_negotiation.cc:407] Connection header received
0928 23:59:19.918202 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:19.918207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:19.918262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:19.918354 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:19.918834 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44166: BlockingRecv error: recv got EOF from 127.0.0.1:44166 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 23:59:20.217373 10867 catalog_manager.cc:1259] Loaded cluster ID: 113932ba486a4be2b428422cf5dd8195
I20260928 23:59:20.217514 10867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: loading cluster ID for follower catalog manager: success
I20260928 23:59:20.218761 10880 catalog_manager.cc:1259] Loaded cluster ID: 113932ba486a4be2b428422cf5dd8195
I20260928 23:59:20.218832 10880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: loading cluster ID for follower catalog manager: success
I20260928 23:59:20.219364 10867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: acquiring CA information for follower catalog manager: success
I20260928 23:59:20.220620 10880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: acquiring CA information for follower catalog manager: success
I20260928 23:59:20.220655 10867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7044eb5a03af4926b3a271be847a82c2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:59:20.221886 10880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b11bcbcff654b2c9a3c0226cf2e702e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:59:20.437537 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:20.433375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.20.88.61:34539)
0928 23:59:20.433655 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:20.433659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:20.433674 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:59:20.433768 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:20.433771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:20.433897 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:20.433993 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:20.434333 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.434881 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.435846 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.436072 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.436176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:20.436565 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:20.437027 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:20.437125 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:20.437256 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:59:20.437314 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260928 23:59:20.445477 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:20.441259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.20.88.60:35865)
0928 23:59:20.441508 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:20.441513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:20.441574 (+ 61us) server_negotiation.cc:407] Connection header received
0928 23:59:20.441890 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:20.441893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:20.442022 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:20.442167 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:20.442423 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.442971 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.443864 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.444110 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.444226 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:20.444577 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:20.445034 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:20.445182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:20.445217 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 23:59:20.445278 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260928 23:59:20.453881 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:20.449050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.20.88.61:34539)
0928 23:59:20.449553 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:20.449558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:20.449573 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:59:20.449638 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:20.449641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:20.449720 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:20.449857 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:20.450135 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.450955 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.452518 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.452891 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.453023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:20.453279 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:20.453372 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:20.453459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:20.453634 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 23:59:20.453700 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260928 23:59:20.454464 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:20.449531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.20.88.60:35865)
0928 23:59:20.449693 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:20.449698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:20.449711 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:59:20.449758 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:20.449764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:20.449833 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:20.449907 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:20.451396 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.451904 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.453099 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.453451 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.453539 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:20.453947 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:20.454024 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:20.454067 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:20.454245 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 23:59:20.454319 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 23:59:20.454957 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:20.448802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.20.88.62:33139)
0928 23:59:20.449065 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:20.449070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:20.449143 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:59:20.449438 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:20.449442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:20.449676 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:20.449829 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:20.450201 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.451008 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.453115 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:20.453469 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:20.453589 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:20.453915 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:20.454321 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:20.454404 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:20.454720 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 23:59:20.454801 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260928 23:59:20.458384 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10617
I20260928 23:59:20.465281 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10692
I20260928 23:59:20.471977 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10767
2026-09-28T23:59:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3200 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:59:20 dist-test-slave-wvpw krb5kdc[10982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:59:20 dist-test-slave-wvpw krb5kdc[10982](info): set up 2 sockets
Sep 28 23:59:20 dist-test-slave-wvpw krb5kdc[10982](info): commencing operation
krb5kdc: starting...
W20260928 23:59:21.533329 20832 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/test-user.keytab.
Sep 28 23:59:21 dist-test-slave-wvpw krb5kdc[10982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.88.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
Entry for principal kudu/127.20.88.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab.
I20260928 23:59:21.672691 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37669
--webserver_interface=127.20.88.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:59:21.788529 10995 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:21.788962 10995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:21.789042 10995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:21.789132 10995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:21.793427 10995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:21.793514 10995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:21.793540 10995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:21.793557 10995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:21.793573 10995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:21.799338 10995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.62.keytab
--principal=kudu/127.20.88.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.62:37669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.88.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:21.800874 10995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:21.801975 10995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:21.811190 11000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:21.811354 11003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:21.811190 11001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:21.811582 10995 server_base.cc:1034] running on GCE node
I20260928 23:59:21.812242 10995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:21.813517 10995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:21.814729 10995 hybrid_clock.cc:648] HybridClock initialized: now 1790639961814713 us; error 27 us; skew 500 ppm
Sep 28 23:59:21 dist-test-slave-wvpw krb5kdc[10982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:21.817834 10995 init.cc:377] Logged in from keytab as kudu/127.20.88.62@KRBTEST.COM (short username kudu)
I20260928 23:59:21.818805 10995 webserver.cc:460] Webserver started at http://127.20.88.62:41041/ using document root <none> and password file <none>
I20260928 23:59:21.819403 10995 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:21.819504 10995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:21.819749 10995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:21.821947 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data/instance:
uuid: "d46c34e0d34347a091b3b144ab783ac1"
format_stamp: "Formatted at 2026-09-28 23:59:21 on dist-test-slave-wvpw"
server_key: "665749d3147aa138ea16969822b300dc"
server_key_iv: "1636d2f6722c3df8354599dfe3f367bf"
server_key_version: "encryptionkey@0"
I20260928 23:59:21.822544 10995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance:
uuid: "d46c34e0d34347a091b3b144ab783ac1"
format_stamp: "Formatted at 2026-09-28 23:59:21 on dist-test-slave-wvpw"
server_key: "665749d3147aa138ea16969822b300dc"
server_key_iv: "1636d2f6722c3df8354599dfe3f367bf"
server_key_version: "encryptionkey@0"
I20260928 23:59:21.826418 10995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:59:21.829141 11010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:21.830747 10995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:59:21.830910 10995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal
uuid: "d46c34e0d34347a091b3b144ab783ac1"
format_stamp: "Formatted at 2026-09-28 23:59:21 on dist-test-slave-wvpw"
server_key: "665749d3147aa138ea16969822b300dc"
server_key_iv: "1636d2f6722c3df8354599dfe3f367bf"
server_key_version: "encryptionkey@0"
I20260928 23:59:21.831039 10995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:21.849139 10995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:21.852722 10995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:21.852937 10995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:21.860168 10995 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.62:37669
I20260928 23:59:21.860543 11062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.62:37669 every 8 connection(s)
I20260928 23:59:21.861287 10995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/data/info.pb
I20260928 23:59:21.864439 11063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:21.868726 11063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:21.870398 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 10995
I20260928 23:59:21.870551 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-0/wal/instance
I20260928 23:59:21.870955 20832 external_mini_cluster.cc:1428] Setting key 4c7d63f93e508b12c03cbcb208992af6
WARNING: no policy specified for kudu/127.20.88.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.61@KRBTEST.COM" created.
Sep 28 23:59:21 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:21.890818 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.871189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.20.88.62:37669)
0928 23:59:21.871814 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:21.871831 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:21.871908 (+ 77us) server_negotiation.cc:407] Connection header received
0928 23:59:21.873509 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:21.873514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:21.873754 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:21.873989 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:21.874867 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:21.876385 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:21.877485 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:21.878118 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:21.878308 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:21.885025 (+ 6717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:21.885036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:21.885056 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:21.885097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:21.887029 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:21.887734 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:21.887740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:21.887741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:21.887857 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:21.888331 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:21.888334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:21.888335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:21.888892 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:21.889003 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:21.889563 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 23:59:21.889807 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":488,"spinlock_wait_cycles":18688,"thread_start_us":161,"threads_started":1}
I20260928 23:59:21.890822 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.871189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37669 (local address 127.0.0.1:35092)
0928 23:59:21.871735 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:21.871794 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:21.873072 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:21.874044 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:21.874052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:21.874431 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:21.874702 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:21.874716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:21.876554 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:21.876558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:21.877339 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:21.877349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:21.878305 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:21.878308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:21.878403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:21.879102 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:21.879121 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:21.884767 (+ 5646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:21.887191 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:21.887207 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:21.887222 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:21.887580 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:21.887963 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:21.887969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:21.887974 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:21.888150 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:21.889026 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:21.889034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:21.889234 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 23:59:21.889447 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 23:59:21.889901 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":64,"thread_start_us":163,"threads_started":1}
I20260928 23:59:21.893321 11063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:21.894141 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.893860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53716)
0928 23:59:21.893991 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:21.894075 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:59:21.894446 11012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
I20260928 23:59:21.895383 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.895174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53724)
0928 23:59:21.895281 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:21.895329 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:59:21.895676 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:39611: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
Entry for principal kudu/127.20.88.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab.
I20260928 23:59:21.904873 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:39611
--webserver_interface=127.20.88.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:59:21.945271 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } attempt: 1
I20260928 23:59:21.946271 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.945907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53738)
0928 23:59:21.946119 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:21.946187 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 23:59:21.947108 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:21.946851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53750)
0928 23:59:21.946968 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:21.947021 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 23:59:21.947243 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:39611: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
I20260928 23:59:22.015446 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } attempt: 2
I20260928 23:59:22.016587 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.016062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53756)
0928 23:59:22.016423 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.016498 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 23:59:22.017395 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.017160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53762)
0928 23:59:22.017283 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.017337 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:59:22.017527 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.61:39611: Network error: Client connection negotiation failed: client connection to 127.20.88.61:39611: connect: Connection refused (error 111)
W20260928 23:59:22.022728 11070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:22.023102 11070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:22.023166 11070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:22.023267 11070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:59:22.027509 11070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:22.027611 11070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:22.027665 11070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:22.027726 11070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:22.027763 11070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:22.033515 11070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.61.keytab
--principal=kudu/127.20.88.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.61:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.88.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:22.034788 11070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:22.035902 11070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:22.044878 11075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:22.044927 11078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:22.044884 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:22.045113 11070 server_base.cc:1034] running on GCE node
I20260928 23:59:22.045838 11070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:22.047031 11070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:22.048267 11070 hybrid_clock.cc:648] HybridClock initialized: now 1790639962048195 us; error 53 us; skew 500 ppm
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:22.051316 11070 init.cc:377] Logged in from keytab as kudu/127.20.88.61@KRBTEST.COM (short username kudu)
I20260928 23:59:22.052351 11070 webserver.cc:460] Webserver started at http://127.20.88.61:38393/ using document root <none> and password file <none>
I20260928 23:59:22.052943 11070 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:22.053040 11070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:22.053287 11070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:22.055449 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data/instance:
uuid: "cfad9dc2268b4e17872f9b83897796e0"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "0ee5a1d6c229b2b27032499925418891"
server_key_iv: "5688c67d1ca35acaab1b8d1fa8aa77de"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.056038 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal/instance:
uuid: "cfad9dc2268b4e17872f9b83897796e0"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "0ee5a1d6c229b2b27032499925418891"
server_key_iv: "5688c67d1ca35acaab1b8d1fa8aa77de"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.059983 11070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 23:59:22.062738 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:22.064172 11070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:59:22.064376 11070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal
uuid: "cfad9dc2268b4e17872f9b83897796e0"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "0ee5a1d6c229b2b27032499925418891"
server_key_iv: "5688c67d1ca35acaab1b8d1fa8aa77de"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.064509 11070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:22.076953 11070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:22.080282 11070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:22.080513 11070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:22.088407 11070 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.61:39611
I20260928 23:59:22.088737 11137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.61:39611 every 8 connection(s)
I20260928 23:59:22.089555 11070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/data/info.pb
I20260928 23:59:22.092046 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 11070
I20260928 23:59:22.092208 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-1/wal/instance
I20260928 23:59:22.092567 20832 external_mini_cluster.cc:1428] Setting key 24cf8bfce80398985a1863b30f6ba2bb
I20260928 23:59:22.093119 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:22.099143 11138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.88.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.88.60@KRBTEST.COM" created.
I20260928 23:59:22.108702 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } attempt: 3
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:59:22.118716 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.101826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.20.88.62:37669)
0928 23:59:22.102002 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.102008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.103162 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 23:59:22.104215 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.104219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.104314 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.104432 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.105900 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.106445 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.107915 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.108363 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.108489 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.111875 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.111885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.111887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.111913 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.115426 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.116210 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.116213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.116215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.116367 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.116791 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.116794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.116795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.117049 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.117129 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.118155 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.118457 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:59:22.119688 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.102319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37669 (local address 127.0.0.1:35108)
0928 23:59:22.102973 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.103040 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.103964 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.104622 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.104636 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.105005 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.105732 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.105751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.106591 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.106597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.107765 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.107778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.108550 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.108554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.108803 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.109872 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.109891 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.111708 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.115602 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.115613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.115638 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.116068 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.116468 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.116470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.116472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.116644 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.117209 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.117215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.117665 (+ 450us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.117908 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.118317 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":202,"threads_started":1}
I20260928 23:59:22.121714 11138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:22.124392 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.109231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53778)
0928 23:59:22.109567 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.109586 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.109691 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.110438 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.110445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.110680 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.110819 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.110827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.112562 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.112568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.113183 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.113192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.114161 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.114168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.114688 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.115146 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.115168 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.118568 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.121966 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.121970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.121972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.122317 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.122700 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.122704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.122706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.122813 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.123450 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.123453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.123719 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.123846 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.124140 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260928 23:59:22.124657 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.109399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.20.88.61:39611)
0928 23:59:22.109830 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.109846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.109875 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:59:22.109928 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.109936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.110211 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.110416 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.110971 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.112424 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.113320 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.113983 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.114140 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.118970 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.118992 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.119013 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.119049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.121789 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.122470 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.122476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.122479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.122578 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.122914 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.122917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.122918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.123316 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.123468 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.124047 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.124320 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":172,"threads_started":1}
I20260928 23:59:22.128916 11063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:59:22.129793 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.129473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34796)
0928 23:59:22.129624 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.129690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 23:59:22.130648 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.130322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34810)
0928 23:59:22.130505 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.130570 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 23:59:22.130829 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
Entry for principal kudu/127.20.88.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab.
I20260928 23:59:22.132799 20832 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHREvfx/build/debug/bin/kudu
/tmp/dist-test-taskHREvfx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:40729
--webserver_interface=127.20.88.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.88.20:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:59:22.134097 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.122316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53782)
0928 23:59:22.122538 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.122553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.122643 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.123662 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.123665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.123882 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.124013 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.124023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.125174 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.125178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.126362 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.126377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.127165 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.127169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.127285 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.127929 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.127948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.129698 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.132368 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.132372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.132376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.132654 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.133037 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.133042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.133044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.133132 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.133531 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.133534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.133623 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.133768 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.133927 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 23:59:22.134408 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.122400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.20.88.61:39611)
0928 23:59:22.123306 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.123316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.123344 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:59:22.123381 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.123384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.123430 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.123518 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.124151 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.125066 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.126477 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.126902 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.127021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.129900 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.129907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.129910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.129936 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.132196 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.132786 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.132789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.132790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.132908 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.133231 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.133235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.133236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.133437 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.133545 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.133827 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.133988 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":791,"thread_start_us":142,"threads_started":1}
I20260928 23:59:22.135092 11138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:22.135815 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.135561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34824)
0928 23:59:22.135687 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.135761 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 23:59:22.136034 11088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
I20260928 23:59:22.136904 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.136696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34826)
0928 23:59:22.136800 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.136845 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:59:22.137122 11138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
I20260928 23:59:22.162055 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 1
I20260928 23:59:22.162997 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.162652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34836)
0928 23:59:22.162845 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.162914 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 23:59:22.163920 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.163692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34850)
0928 23:59:22.163813 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.163860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 23:59:22.164175 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
I20260928 23:59:22.186445 11138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 1
I20260928 23:59:22.187491 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.187156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34864)
0928 23:59:22.187339 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.187408 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 23:59:22.188400 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.188114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34870)
0928 23:59:22.188281 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.188340 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 23:59:22.188544 11138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
I20260928 23:59:22.239441 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 2
I20260928 23:59:22.240417 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.240031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34884)
0928 23:59:22.240190 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.240321 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 23:59:22.241165 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.240939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34898)
0928 23:59:22.241048 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.241110 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:59:22.241290 11063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
W20260928 23:59:22.252569 11145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:59:22.252925 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:59:22.252987 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:59:22.253082 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:59:22.256717 11138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 2
I20260928 23:59:22.257550 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.257254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34902)
0928 23:59:22.257402 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.257474 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 23:59:22.257655 11145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:59:22.257755 11145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:59:22.257807 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:59:22.257846 11145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:59:22.257915 11145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:59:22.258299 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.258061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34904)
0928 23:59:22.258187 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.258247 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:59:22.258428 11138 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.88.60:40729: Network error: Client connection negotiation failed: client connection to 127.20.88.60:40729: connect: Connection refused (error 111)
I20260928 23:59:22.263895 11145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.88.20:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.88.62:37669,127.20.88.61:39611,127.20.88.60:40729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/krb5kdc/kudu_127.20.88.60.keytab
--principal=kudu/127.20.88.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.88.60:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.88.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:59:22.265172 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:59:22.266250 11145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:59:22.275475 11151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:22.275940 11152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:59:22.275983 11154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:59:22.276127 11145 server_base.cc:1034] running on GCE node
I20260928 23:59:22.276717 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:59:22.277926 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:59:22.279150 11145 hybrid_clock.cc:648] HybridClock initialized: now 1790639962279132 us; error 32 us; skew 500 ppm
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:59:22.282298 11145 init.cc:377] Logged in from keytab as kudu/127.20.88.60@KRBTEST.COM (short username kudu)
I20260928 23:59:22.283411 11145 webserver.cc:460] Webserver started at http://127.20.88.60:42965/ using document root <none> and password file <none>
I20260928 23:59:22.284003 11145 fs_manager.cc:359] Metadata directory not provided
I20260928 23:59:22.284097 11145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:59:22.284390 11145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:59:22.286502 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data/instance:
uuid: "e1ae0b4115d448de936e1e8c12d72593"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "28f70a2328fb0f30f85b865f0396fbd2"
server_key_iv: "087e9be723b3337b1eca1ce2cfc5e2f3"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.287168 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal/instance:
uuid: "e1ae0b4115d448de936e1e8c12d72593"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "28f70a2328fb0f30f85b865f0396fbd2"
server_key_iv: "087e9be723b3337b1eca1ce2cfc5e2f3"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.291389 11145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:59:22.294243 11161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:59:22.295789 11145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:59:22.295961 11145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data,/tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal
uuid: "e1ae0b4115d448de936e1e8c12d72593"
format_stamp: "Formatted at 2026-09-28 23:59:22 on dist-test-slave-wvpw"
server_key: "28f70a2328fb0f30f85b865f0396fbd2"
server_key_iv: "087e9be723b3337b1eca1ce2cfc5e2f3"
server_key_version: "encryptionkey@0"
I20260928 23:59:22.296103 11145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:59:22.310050 11145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:59:22.313921 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:59:22.314181 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:59:22.322158 11145 rpc_server.cc:307] RPC server started. Bound to: 127.20.88.60:40729
I20260928 23:59:22.322465 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.88.60:40729 every 8 connection(s)
I20260928 23:59:22.323280 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/data/info.pb
I20260928 23:59:22.326236 11214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:59:22.330533 20832 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHREvfx/build/debug/bin/kudu as pid 11145
I20260928 23:59:22.330199 11214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:22.330674 20832 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHREvfx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639669102918-20832-0/minicluster-data/master-2/wal/instance
I20260928 23:59:22.330974 20832 external_mini_cluster.cc:1428] Setting key 02dd200902d1251ad271ac7529bcd1f8
I20260928 23:59:22.340482 11063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 3
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.62@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:59:22.346654 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.332411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.20.88.62:37669)
0928 23:59:22.332567 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.332571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.333175 (+ 604us) server_negotiation.cc:407] Connection header received
0928 23:59:22.333967 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.333970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.334019 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.334119 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.335394 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.336255 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.337806 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.338155 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.338288 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.342218 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.342222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.342224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.342250 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.343905 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.344639 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.344642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.344644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.344720 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.345216 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.345220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.345222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.345478 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.345597 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.346313 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.346451 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:59:22.347771 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.332505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.62:37669 (local address 127.0.0.1:35110)
0928 23:59:22.333008 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.333060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.333751 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.334209 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.334214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.334845 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.335255 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.335270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.336400 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.336403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.337624 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.337634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.338387 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.338390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.338560 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.339477 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.339501 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.342013 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.344037 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.344050 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.344066 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.344498 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.344833 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.344835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.344838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.345077 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.345629 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.345637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.345995 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.346248 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.346617 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"spinlock_wait_cycles":768,"thread_start_us":143,"threads_started":1}
I20260928 23:59:22.348148 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.332504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.20.88.60:40729)
0928 23:59:22.332961 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.332975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.333008 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:59:22.333729 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.333749 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.334001 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.334177 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.335215 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.336526 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.337805 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.338434 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.338555 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.342494 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.342508 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.342528 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.342557 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.345632 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.346095 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.346104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.346108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.346197 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.346531 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.346535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.346536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.346842 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.347066 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.347312 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.347479 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"spinlock_wait_cycles":9472,"thread_start_us":176,"threads_started":1}
I20260928 23:59:22.349129 11214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:22.349684 11138 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } attempt: 3
I20260928 23:59:22.350800 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.341011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34920)
0928 23:59:22.341151 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.341168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.341253 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.341744 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.341747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.341956 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.342047 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.342052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.343068 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.343072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.343742 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.343752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.344414 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.344417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.344509 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.344978 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.344992 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.346407 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.349100 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.349114 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.349116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.349380 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.349728 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.349731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.349733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.349829 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.350286 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.350289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.350394 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.350479 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.350628 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 23:59:22.351462 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.341140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.20.88.60:40729)
0928 23:59:22.341522 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.341528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.341539 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:59:22.341572 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.341574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.341618 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.341723 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.342170 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.342932 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.343884 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.344288 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.344407 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.346551 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.346557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.346562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.346588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.348974 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.349520 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.349526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.349530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.349620 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.349944 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.349949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.349951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.350178 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.351077 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.351166 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.351290 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"spinlock_wait_cycles":1280,"thread_start_us":144,"threads_started":1}
I20260928 23:59:22.355391 11063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: Bootstrap starting.
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.61@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.62@KRBTEST.COM
I20260928 23:59:22.358624 11063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:22.359589 11063 log.cc:824] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:22.362284 11063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: No bootstrap required, opened a new log
I20260928 23:59:22.362391 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.349946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.20.88.61:39611)
0928 23:59:22.350362 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.350367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.350380 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:59:22.350414 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.350417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.350461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.350549 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.351218 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.351893 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.353172 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.353546 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.353616 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.358286 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.358289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.358292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.358317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.360040 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.360773 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.360780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.360784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.360872 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361228 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.361232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.361234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.361566 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.361651 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.362024 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.362189 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":342}
I20260928 23:59:22.362620 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.350182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34922)
0928 23:59:22.350304 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.350319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.350389 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.351424 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.351429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.351626 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.351751 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.351759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.353220 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.353223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.354300 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.354312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.355138 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.355144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.355247 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.355712 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.355733 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.357670 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.361043 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.361048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.361305 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.361621 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.361626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.361712 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.362158 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.362161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.362255 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.362340 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.362477 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":302}
I20260928 23:59:22.362896 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.349656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.61:39611 (local address 127.0.0.1:53788)
0928 23:59:22.349907 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.349921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.349986 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.350553 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.350556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.350733 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.351086 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.351096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.352145 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.352148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.353036 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.353044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.354477 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.354480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.354589 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.355078 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.355096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.358143 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.360344 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.360351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.360356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.360650 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.360999 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.361010 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.361115 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.361711 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.361715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.361836 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.362543 (+ 707us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.362698 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":20}
I20260928 23:59:22.363149 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.350225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.20.88.60:40729)
0928 23:59:22.351208 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.351214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.351228 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:59:22.351260 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.351262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.351297 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.352324 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.352403 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.353112 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.354589 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.354985 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.355463 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.357837 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.357841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.357844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.357871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.360918 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361430 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.361433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.361435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.361514 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361822 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.361824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.361826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.362030 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.362785 (+ 755us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.362861 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.362974 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":916}
I20260928 23:59:22.363739 11214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:59:22.366432 11138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: Bootstrap starting.
I20260928 23:59:22.366770 11063 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.366991 11063 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46c34e0d34347a091b3b144ab783ac1, State: Initialized, Role: FOLLOWER
I20260928 23:59:22.367348 11063 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.367983 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.368254 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.369253 11063 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:22.369398 11138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: Neither blocks nor log segments found. Creating new log.
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639962, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.88.60@KRBTEST.COM for kudu/127.20.88.60@KRBTEST.COM
I20260928 23:59:22.370386 11138 log.cc:824] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:59:22.371208 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.350814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.20.88.62:37669)
0928 23:59:22.350919 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.350923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.350996 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:59:22.351141 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.351144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.351188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.351259 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.352116 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.352826 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.353902 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.354245 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.354376 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.358025 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.358028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.358031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.358061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.360515 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361083 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.361087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.361088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.361166 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.361470 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.361473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.361475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.361711 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.361877 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.370876 (+ 8999us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.371032 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:59:22.373076 11138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: No bootstrap required, opened a new log
I20260928 23:59:22.374231 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.364306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.88.60:40729 (local address 127.0.0.1:34938)
0928 23:59:22.364493 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:59:22.364508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:59:22.364580 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:59:22.364841 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:59:22.364844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:59:22.365008 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:59:22.365114 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.365120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.366119 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.366122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.366899 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:59:22.366909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.367777 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.367780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:59:22.367861 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.368450 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:59:22.368464 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:59:22.370189 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:59:22.372662 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.372667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.372671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.372915 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.373222 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:59:22.373225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:59:22.373226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:59:22.373313 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:59:22.373742 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:59:22.373744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:59:22.373821 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 23:59:22.373913 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:59:22.374049 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 23:59:22.374717 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.364433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.20.88.60:40729)
0928 23:59:22.364538 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.364542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.364556 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:59:22.364679 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.364682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.364724 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.364806 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.365237 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.365989 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.367185 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.367521 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.367628 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.370354 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.370358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.370360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.370384 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.372542 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.373015 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.373018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.373020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.373094 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.373407 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.373410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.373412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.373619 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.373710 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.374446 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.374563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:59:22.378252 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: Bootstrap starting.
W20260928 23:59:22.378552 11233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:22.378641 11233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:22.378459 11138 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.378911 11138 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfad9dc2268b4e17872f9b83897796e0, State: Initialized, Role: FOLLOWER
Sep 28 23:59:22 dist-test-slave-wvpw krb5kdc[10982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.88.61@KRBTEST.COM
I20260928 23:59:22.379381 11138 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.380944 11138 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:22.381371 11214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: Neither blocks nor log segments found. Creating new log.
I20260928 23:59:22.382367 11214 log.cc:824] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: Log is configured to *not* fsync() on all Append() calls
W20260928 23:59:22.387128 11247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:22.388208 11247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:22.388069 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.388471 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.387811 10995 master_runner.cc:186] Error getting master registration for 127.20.88.61:39611: OK, instance_id { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" instance_seqno: 1790639962086170 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:59:22.392138 10995 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:59:22.384748 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.373212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.20.88.61:39611)
0928 23:59:22.373324 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.373329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.373342 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:59:22.373379 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.373382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.373425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.373504 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.374414 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.375196 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.376364 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.376674 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.376773 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.379905 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.379909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.379912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.379942 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.381785 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.382358 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.382362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.382364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.382456 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.382778 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.382781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.382783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.383003 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.383710 (+ 707us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.384438 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.384563 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:59:22.394795 11214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: No bootstrap required, opened a new log
I20260928 23:59:22.396059 20832 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:59:22.397354 11037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d46c34e0d34347a091b3b144ab783ac1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35112
I20260928 23:59:22.397622 11037 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:59:22.397749 11037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:22.399308 11037 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.399787 11214 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.400278 11214 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ae0b4115d448de936e1e8c12d72593, State: Initialized, Role: FOLLOWER
I20260928 23:59:22.400743 11214 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.401208 11112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46c34e0d34347a091b3b144ab783ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cfad9dc2268b4e17872f9b83897796e0"
I20260928 23:59:22.401480 11112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:22.398922 11070 master_runner.cc:186] Error getting master registration for 127.20.88.60:40729: OK, instance_id { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" instance_seqno: 1790639962320018 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:59:22.402473 11070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:59:22.402786 11214 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:59:22.403045 11112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46c34e0d34347a091b3b144ab783ac1 in term 1.
I20260928 23:59:22.403558 11012 leader_election.cc:302] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfad9dc2268b4e17872f9b83897796e0, d46c34e0d34347a091b3b144ab783ac1; no voters:
I20260928 23:59:22.403942 11221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:59:22.404217 11221 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [term 1 LEADER]: Becoming Leader. State: Replica: d46c34e0d34347a091b3b144ab783ac1, State: Running, Role: LEADER
I20260928 23:59:22.404264 11037 leader_election.cc:288] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [CANDIDATE]: Term 1 election: Requested vote from peers cfad9dc2268b4e17872f9b83897796e0 (127.20.88.61:39611), e1ae0b4115d448de936e1e8c12d72593 (127.20.88.60:40729)
I20260928 23:59:22.404675 11221 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } }
I20260928 23:59:22.404717 11188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d46c34e0d34347a091b3b144ab783ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1ae0b4115d448de936e1e8c12d72593"
I20260928 23:59:22.404950 11188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:59:22.406747 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46c34e0d34347a091b3b144ab783ac1. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.406819 11188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d46c34e0d34347a091b3b144ab783ac1 in term 1.
I20260928 23:59:22.406868 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: This master's current role is: LEADER
W20260928 23:59:22.417192 11267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:59:22.417364 11267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:59:22.422945 11258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:59:22.424000 11258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:59:22.427484 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.411384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.20.88.61:39611)
0928 23:59:22.411501 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.411506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.416507 (+ 5001us) server_negotiation.cc:407] Connection header received
0928 23:59:22.416694 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.416697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.416750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.416803 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.417897 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.418762 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.420066 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.420479 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.420560 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.423438 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.423443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.423446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.423477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.425243 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.426016 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.426020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.426022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.426133 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.426447 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.426451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.426453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.426686 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.426791 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.427179 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.427312 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:59:22.427870 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.428035 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.434038 11188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [term 1 FOLLOWER]: Refusing update from remote peer d46c34e0d34347a091b3b144ab783ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:59:22.434870 11252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:59:22.440003 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46c34e0d34347a091b3b144ab783ac1. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.440183 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.440428 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.412089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.20.88.60:40729)
0928 23:59:22.412213 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.412218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.417709 (+ 5491us) server_negotiation.cc:407] Connection header received
0928 23:59:22.417882 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.417885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.417930 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.417979 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.418935 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.419700 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.420915 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.421277 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.421347 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.436261 (+ 14914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.436265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.436269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.436306 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.438151 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.438718 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.438722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.438724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.438809 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.439121 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.439124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.439126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.439362 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.439479 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.439821 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.439952 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:59:22.442958 11112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [term 1 FOLLOWER]: Refusing update from remote peer d46c34e0d34347a091b3b144ab783ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:59:22.443665 11252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:59:22.443920 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:22.410646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.20.88.62:37669)
0928 23:59:22.410771 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.410776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.412777 (+ 2001us) server_negotiation.cc:407] Connection header received
0928 23:59:22.412977 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.412981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.413032 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.413085 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.414188 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.415078 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.424515 (+ 9437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:22.424907 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:22.424992 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:22.430194 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:59:22.430199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:59:22.430202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:59:22.430231 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:59:22.432093 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.442317 (+ 10224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.442322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.442324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.442440 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:59:22.442787 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:59:22.442791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:59:22.442793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:59:22.443093 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:59:22.443231 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:22.443624 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 23:59:22.443767 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:59:22.448251 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46c34e0d34347a091b3b144ab783ac1. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.448427 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.459009 11252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.459180 11252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:22.459323 11272 mvcc.cc:205] Tried to move back new op lower bound from 7334461286113222656 to 7334461286013652992. Current Snapshot: MvccSnapshot[applied={T|T < 7334461286113222656}]
I20260928 23:59:22.461117 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.461264 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.461874 11275 mvcc.cc:205] Tried to move back new op lower bound from 7334461286113222656 to 7334461286013652992. Current Snapshot: MvccSnapshot[applied={T|T < 7334461286113222656}]
I20260928 23:59:22.462874 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.463019 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:59:22.463413 11258 catalog_manager.cc:1347] Generated new cluster ID: f8b880cdae7449ecabecb0ed8c58fe95
I20260928 23:59:22.463479 11258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:59:22.464334 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d46c34e0d34347a091b3b144ab783ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d46c34e0d34347a091b3b144ab783ac1" member_type: VOTER last_known_addr { host: "127.20.88.62" port: 37669 } } peers { permanent_uuid: "cfad9dc2268b4e17872f9b83897796e0" member_type: VOTER last_known_addr { host: "127.20.88.61" port: 39611 } } peers { permanent_uuid: "e1ae0b4115d448de936e1e8c12d72593" member_type: VOTER last_known_addr { host: "127.20.88.60" port: 40729 } } }
I20260928 23:59:22.464478 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:59:22.465111 11273 mvcc.cc:205] Tried to move back new op lower bound from 7334461286113222656 to 7334461286013652992. Current Snapshot: MvccSnapshot[applied={T|T < 7334461286113222656}]
I20260928 23:59:22.486493 11258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:59:22.488020 11258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:59:22.513978 11258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46c34e0d34347a091b3b144ab783ac1: Generated new TSK 0
I20260928 23:59:22.515112 11258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:59:22.549708 11218 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.546156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.20.88.60:40729)
0928 23:59:22.546304 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.546309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.548331 (+ 2022us) server_negotiation.cc:407] Connection header received
0928 23:59:22.548547 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.548551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.548603 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.548657 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.549370 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34962: BlockingRecv error: recv got EOF from 127.0.0.1:34962 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:59:22.551783 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.547675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.20.88.61:39611)
0928 23:59:22.547790 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.547794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.550481 (+ 2687us) server_negotiation.cc:407] Connection header received
0928 23:59:22.550744 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.550748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.550804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.550857 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.551655 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53810: BlockingRecv error: recv got EOF from 127.0.0.1:53810 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 23:59:22.552793 11066 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.547009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.20.88.62:37669)
0928 23:59:22.547128 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.547132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.550304 (+ 3172us) server_negotiation.cc:407] Connection header received
0928 23:59:22.550993 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.550997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.551048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.551098 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.552670 (+ 1572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35124: BlockingRecv error: recv got EOF from 127.0.0.1:35124 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 23:59:22.558284 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.556327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.20.88.61:39611)
0928 23:59:22.556449 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.556454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.556961 (+ 507us) server_negotiation.cc:407] Connection header received
0928 23:59:22.557210 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.557215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.557299 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.557356 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.558137 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53822: BlockingRecv error: recv got EOF from 127.0.0.1:53822 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:59:22.564599 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.562631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.20.88.61:39611)
0928 23:59:22.562762 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.562767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.563376 (+ 609us) server_negotiation.cc:407] Connection header received
0928 23:59:22.563592 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.563597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.563657 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.563717 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.564466 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53830: BlockingRecv error: recv got EOF from 127.0.0.1:53830 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:59:22.571749 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.569835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.20.88.61:39611)
0928 23:59:22.569968 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.569974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.570628 (+ 654us) server_negotiation.cc:407] Connection header received
0928 23:59:22.570861 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.570865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.570929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.570987 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.571623 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53842: BlockingRecv error: recv got EOF from 127.0.0.1:53842 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:59:22.580644 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.578703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.20.88.61:39611)
0928 23:59:22.578868 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.578874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.579492 (+ 618us) server_negotiation.cc:407] Connection header received
0928 23:59:22.579717 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.579721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.579780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.579839 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.580509 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53856: BlockingRecv error: recv got EOF from 127.0.0.1:53856 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 23:59:22.593887 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.591844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.20.88.61:39611)
0928 23:59:22.591987 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.591992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.592732 (+ 740us) server_negotiation.cc:407] Connection header received
0928 23:59:22.592965 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.592969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.593036 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.593108 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.593742 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53872: BlockingRecv error: recv got EOF from 127.0.0.1:53872 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:59:22.615409 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.613280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.20.88.61:39611)
0928 23:59:22.613427 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.613432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.614161 (+ 729us) server_negotiation.cc:407] Connection header received
0928 23:59:22.614451 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.614456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.614526 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.614594 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.615278 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53874: BlockingRecv error: recv got EOF from 127.0.0.1:53874 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:59:22.651417 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.650266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.20.88.61:39611)
0928 23:59:22.650408 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.650413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.650577 (+ 164us) server_negotiation.cc:407] Connection header received
0928 23:59:22.650695 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.650697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.650745 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.650817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.651272 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53886: BlockingRecv error: recv got EOF from 127.0.0.1:53886 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 23:59:22.719182 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.717847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.20.88.61:39611)
0928 23:59:22.718036 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.718042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.718220 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:59:22.718413 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.718416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.718473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.718570 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.719069 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53902: BlockingRecv error: recv got EOF from 127.0.0.1:53902 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 23:59:22.851012 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:22.849719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.20.88.61:39611)
0928 23:59:22.849898 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:22.849903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:22.850043 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:59:22.850246 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:22.850250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:22.850306 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:22.850401 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:22.850875 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53908: BlockingRecv error: recv got EOF from 127.0.0.1:53908 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 23:59:23.110795 11144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:59:23.109683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.20.88.61:39611)
0928 23:59:23.109847 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.109854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.109871 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:59:23.110040 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.110044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.110121 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.110218 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:59:23.110673 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53920: BlockingRecv error: recv got EOF from 127.0.0.1:53920 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:59:23.390166 11247 catalog_manager.cc:1259] Loaded cluster ID: f8b880cdae7449ecabecb0ed8c58fe95
I20260928 23:59:23.390285 11247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: loading cluster ID for follower catalog manager: success
I20260928 23:59:23.392072 11247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: acquiring CA information for follower catalog manager: success
I20260928 23:59:23.393428 11247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfad9dc2268b4e17872f9b83897796e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:59:23.418867 11267 catalog_manager.cc:1259] Loaded cluster ID: f8b880cdae7449ecabecb0ed8c58fe95
I20260928 23:59:23.419096 11267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: loading cluster ID for follower catalog manager: success
I20260928 23:59:23.421115 11267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: acquiring CA information for follower catalog manager: success
I20260928 23:59:23.422420 11267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1ae0b4115d448de936e1e8c12d72593: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:59:23.629925 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:23.625379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.20.88.61:39611)
0928 23:59:23.625630 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.625633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.625716 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:59:23.625835 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.625838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.625969 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.626069 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:23.626356 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.626908 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.628105 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.628402 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.628497 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:23.628913 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:23.629405 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:23.629505 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:23.629678 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:59:23.629741 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260928 23:59:23.638264 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:23.633926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.20.88.60:40729)
0928 23:59:23.634156 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.634160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.634283 (+ 123us) server_negotiation.cc:407] Connection header received
0928 23:59:23.634416 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.634419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.634658 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.634773 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:23.635035 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.635556 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.636670 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.636896 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.636973 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:23.637290 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:23.637737 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:23.637828 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:23.638040 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:59:23.638110 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260928 23:59:23.646883 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:23.642257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.20.88.61:39611)
0928 23:59:23.642400 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.642405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.642877 (+ 472us) server_negotiation.cc:407] Connection header received
0928 23:59:23.643101 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.643105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.643180 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.643259 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:23.643693 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.644295 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.645663 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.646038 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.646154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:23.646385 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:23.646471 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:23.646551 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:23.646676 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:59:23.646738 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:59:23.647235 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:23.641866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.20.88.62:37669)
0928 23:59:23.642309 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.642314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.642328 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:59:23.642548 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.642551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.642735 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.642857 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:23.643220 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.644184 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.645329 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.645653 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.645765 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:23.646120 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:23.646514 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:23.646610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:23.646936 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 23:59:23.647018 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260928 23:59:23.647845 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:59:23.642063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.20.88.60:40729)
0928 23:59:23.642223 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:59:23.642227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:59:23.642373 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:59:23.642607 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:59:23.642610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:59:23.642672 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:59:23.642748 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:59:23.643258 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.644156 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.646235 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:59:23.646601 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:59:23.646724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:59:23.647292 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:59:23.647390 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:59:23.647449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:59:23.647650 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:59:23.647712 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:59:23.655179 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 10995
I20260928 23:59:23.663759 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 11070
I20260928 23:59:23.670382 20832 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHREvfx/build/debug/bin/kudu with pid 11145
2026-09-28T23:59:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3198 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6398 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294559 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST